Jul 09 17:39:45.218099 ip-10-0-136-100 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 17:39:46.697624 ip-10-0-136-100 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:46.697624 ip-10-0-136-100 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 17:39:46.697624 ip-10-0-136-100 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:46.697624 ip-10-0-136-100 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 17:39:46.697624 ip-10-0-136-100 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:46.698298 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.697713    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 17:39:46.702111 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702089    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.702111 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702109    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.702111 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702113    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702117    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702120    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702123    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702126    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702129    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702133    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702136    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702139    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702142    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702144    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702153    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702156    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702158    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702161    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702164    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702166    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702169    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702172    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702174    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.702221 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702177    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702180    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702183    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702186    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702192    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702195    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702197    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702200    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702202    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702205    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702207    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702210    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702213    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702216    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702219    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702221    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702225    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702228    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702230    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702233    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.702709 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702236    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702238    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702241    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702244    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702246    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702249    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702251    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702254    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702256    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702259    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702262    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702264    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702267    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702269    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702272    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702277    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702280    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702283    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702286    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.703188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702289    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702292    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702295    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702297    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702300    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702305    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702307    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702311    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702313    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702316    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702319    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702322    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702324    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702327    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702329    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702332    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702335    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702337    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702340    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702342    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.703675 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702345    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702348    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702350    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702354    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702356    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702370    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702373    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702376    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702380    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702383    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702386    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702390    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702392    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702395    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702397    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702400    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702403    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.702406    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703095    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703102    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.704169 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703105    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703108    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703111    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703117    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703120    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703123    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703125    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703128    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703131    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703133    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703136    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703138    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703141    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703144    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703146    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703149    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703152    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703154    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703157    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.704695 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703160    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703163    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703166    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703168    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703171    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703173    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703176    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703178    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703183    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703186    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703191    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703194    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703198    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703201    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703204    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703207    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703210    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703212    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703215    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.705153 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703218    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703220    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703223    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703226    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703228    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703231    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703233    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703236    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703238    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703241    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703243    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703246    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703248    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703251    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703253    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703256    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703258    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703261    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703263    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703265    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.705665 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703268    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703270    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703273    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703275    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703279    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703284    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703287    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703289    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703292    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703294    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703297    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703301    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703303    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703306    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703308    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703311    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703313    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703316    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703318    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703321    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.706156 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703324    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703326    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703329    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703331    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703333    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703336    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703339    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703342    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703344    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703347    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703349    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703352    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703355    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703372    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703375    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703380    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703383    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703386    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703389    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.706640 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703737    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703742    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703745    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703748    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703751    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703754    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703757    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703760    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703763    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703765    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703768    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703770    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703773    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703775    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703778    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703780    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703783    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703785    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703788    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703790    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.707098 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703793    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703796    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703798    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703801    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703804    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703806    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703809    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703812    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703815    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703818    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703820    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703823    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703826    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703828    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703831    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703833    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703836    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703839    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703841    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703843    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.707672 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703846    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703849    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703851    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703854    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703856    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703859    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703861    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703865    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703867    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703869    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703872    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703874    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703877    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703880    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703882    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703885    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703887    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703889    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703892    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703894    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.708175 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703897    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703900    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703902    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703905    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703907    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703910    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703912    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703915    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703917    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703920    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703922    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703925    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703927    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703930    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703932    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703935    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703938    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703941    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703943    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.708684 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703946    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703949    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703951    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703954    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703958    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703964    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703967    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703971    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703975    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703977    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703980    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703983    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703986    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703989    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703991    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703994    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703996    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.703999    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704002    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.709157 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704004    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704077    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704084    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704100    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704104    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704111    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704114    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704118    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704123    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704126    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704129    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704133    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704136    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704139    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704142    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704146    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704149    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704151    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704156    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704160    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704163    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704166    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704169    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704173    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 17:39:46.709645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704176    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704180    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704183    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704187    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704190    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704193    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704196    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704199    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704202    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704206    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704209    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704212    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704215    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704218    2577 flags.go:64] FLAG: --enable-server="true"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704221    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704226    2577 flags.go:64] FLAG: --event-burst="100"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704229    2577 flags.go:64] FLAG: --event-qps="50"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704232    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704235    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704238    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704242    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704244    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704251    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704254    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704257    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 17:39:46.710234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704259    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704264    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704267    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704270    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704272    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704275    2577 flags.go:64] FLAG: --feature-gates=""
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704279    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704282    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704286    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704289    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704292    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704294    2577 flags.go:64] FLAG: --help="false"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704298    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-100.ec2.internal"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704301    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704304    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704307    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704310    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704313    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704316    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704319    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704322    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704325    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704328    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704332    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 17:39:46.710891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704335    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704338    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704340    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704343    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704346    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704350    2577 flags.go:64] FLAG: --lock-file=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704353    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704356    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704374    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704381    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704384    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704387    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704390    2577 flags.go:64] FLAG: --logging-format="text"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704393    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704396    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704399    2577 flags.go:64] FLAG: --manifest-url=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704402    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704407    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704410    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704414    2577 flags.go:64] FLAG: --max-pods="110"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704417    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704420    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704423    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704426    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704429    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 17:39:46.711505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704432    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704435    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704441    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704444    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704447    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704450    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704454    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704459    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704462    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704465    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704469    2577 flags.go:64] FLAG: --port="10250"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704472    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704475    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f5a77f867abd4f2a"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704479    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704482    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704485    2577 flags.go:64] FLAG: --register-node="true"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704489    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704493    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704496    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704498    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704501    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704505    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704508    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704511    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704514    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 17:39:46.712116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704517    2577 flags.go:64] FLAG: --runonce="false"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704520    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704523    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704526    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704529    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704531    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704534    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704538    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704541    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704544    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704546    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704549    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704552    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704555    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704558    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704561    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704566    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704569    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704572    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704575    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704580    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704583    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704586    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704590    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704594    2577 flags.go:64] FLAG: --v="2"
Jul 09 17:39:46.712739 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704598    2577 flags.go:64] FLAG: --version="false"
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704601    2577 flags.go:64] FLAG: --vmodule=""
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704605    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.704608    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704716    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704720    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704723    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704727    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704729    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704732    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704734    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704737    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704739    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704742    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704745    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704747    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704750    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704752    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704755    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704758    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704762    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.713334 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704765    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704768    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704771    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704774    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704776    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704778    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704782    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704785    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704787    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704792    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704795    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704797    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704800    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704802    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704805    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704807    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704810    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704812    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704815    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.713898 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704818    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704820    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704823    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704825    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704828    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704830    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704833    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704835    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704839    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704843    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704845    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704849    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704852    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704856    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704859    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704862    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704865    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704868    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704870    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.714404 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704875    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704878    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704880    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704884    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704887    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704890    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704893    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704896    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704898    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704901    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704903    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704906    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704909    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704911    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704914    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704917    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704919    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704922    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704924    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704927    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.714857 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704929    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704932    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704934    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704937    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704939    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704942    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704944    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704947    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704949    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704952    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704955    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704957    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704961    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704964    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704967    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704971    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704974    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704977    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704979    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704982    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.715344 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704984    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704987    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704989    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.704992    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.705678    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.712435    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.712452    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.712467    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.712473    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712528    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712533    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712537    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712540    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712543    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.715830 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712546    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712549    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712552    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712555    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712557    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712560    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712563    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712565    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712568    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712571    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712574    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712576    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712578    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712581    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712585    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712589    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712592    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712595    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712597    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712600    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.716188 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712603    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712606    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712608    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712611    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712615    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712619    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712622    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712625    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712627    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712630    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712633    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712635    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712638    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712640    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712643    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712646    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712648    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712651    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712653    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712656    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.716680 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712659    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712661    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712664    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712666    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712669    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712671    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712674    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712677    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712680    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712682    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712684    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712687    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712689    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712692    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712694    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712697    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712700    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712704    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712706    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.717152 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712709    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712711    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712716    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712719    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712722    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712726    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712729    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712731    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712734    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712737    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712739    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712742    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712745    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712748    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712751    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712753    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712756    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712759    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712761    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.717905 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712764    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712766    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712769    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712771    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712774    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712776    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712779    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712782    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712785    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712788    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712792    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712795    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712798    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712801    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712804    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.718481 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712807    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.712812    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712948    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712953    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712956    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712959    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712962    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712965    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712970    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712974    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712977    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712980    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712982    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712985    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712987    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:46.718900 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712990    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712993    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712995    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.712998    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713000    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713003    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713006    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713008    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713011    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713013    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713016    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713018    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713021    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713024    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713027    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713030    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713032    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713035    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713038    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:46.719462 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713040    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713043    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713046    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713049    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713051    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713053    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713056    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713058    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713061    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713063    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713066    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713068    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713071    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713073    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713076    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713078    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713081    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713083    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713086    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:46.719927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713088    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713091    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713093    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713096    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713098    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713100    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713103    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713106    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713110    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713114    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713118    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713120    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713123    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713125    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713128    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713130    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713133    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713135    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713138    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713140    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:46.720428 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713143    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713145    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713147    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713150    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713152    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713155    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713157    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713159    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713162    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713166    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713169    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713172    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713175    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713177    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713179    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713182    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713184    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713187    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713189    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713192    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:46.720915 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713195    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713197    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713199    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713202    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713205    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713207    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713210    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:46.713212    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.713217    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.713340    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.716178    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.718885    2577 server.go:1019] "Starting client certificate rotation"
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.719007    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:46.721425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.719218    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:46.721768 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.721674    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:46.722843 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.722822    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:46.731087 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.731060    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 09 17:39:46.733555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.733538    2577 log.go:25] "Validated CRI v1 image API"
Jul 09 17:39:46.734347 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.734326    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 17:39:46.737275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.737254    2577 fs.go:135] Filesystem UUIDs: map[1f9ce1a7-81bb-46fa-9e75-d4ebc964d2fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3984081-8c92-438c-88d8-7b328efd833a:/dev/nvme0n1p3]
Jul 09 17:39:46.737347 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.737274    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 17:39:46.743191 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.743082    2577 manager.go:216] Machine: {Timestamp:2026-07-09 17:39:46.741658041 +0000 UTC m=+0.344543508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092087 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c7c65566a41d3ecb401ce63438947 SystemUUID:ec2c7c65-566a-41d3-ecb4-01ce63438947 BootID:a4a06946-8da6-4abd-8b2b-a6c4bbd07882 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:84:31:85:03:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:84:31:85:03:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:5c:fb:af:dc:c9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 17:39:46.743191 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.743187    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 17:39:46.743294 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.743263    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 17:39:46.746024 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.745995    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 17:39:46.746171 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.746028    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 17:39:46.746794 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.746784    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 17:39:46.746824 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.746797    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 17:39:46.746824 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.746804    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:46.747464 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.747453    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:46.747874 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.747860    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:46.748915 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.748904    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 17:39:46.752208 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752197    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:46.752320 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752311    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 17:39:46.752350 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752338    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 17:39:46.752350 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752348    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 17:39:46.752428 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752372    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 17:39:46.752428 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752381    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 09 17:39:46.752428 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752390    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 17:39:46.752733 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752721    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:46.752768 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.752742    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:46.755126 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.755112    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 17:39:46.757878 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.757859    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 17:39:46.757959 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.757914    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 17:39:46.758111 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758099    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 17:39:46.758148 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758117    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 17:39:46.758148 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758128    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 17:39:46.758148 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758134    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 17:39:46.758148 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758143    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 17:39:46.758255 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758150    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 17:39:46.758677 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758668    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 17:39:46.758712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758678    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 17:39:46.758712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758692    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 17:39:46.758712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758699    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 17:39:46.758712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758708    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 17:39:46.758814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.758719    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 17:39:46.760491 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.760441    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 17:39:46.760491 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.760468    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 17:39:46.763233 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.763215    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 17:39:46.763806 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.763783    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 17:39:46.763862 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.763787    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 17:39:46.764113 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.764099    2577 server.go:1268] "Started kubelet"
Jul 09 17:39:46.764189 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.764152    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 17:39:46.764234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.764178    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 17:39:46.764299 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.764239    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 17:39:46.765106 ip-10-0-136-100 systemd[1]: Started Kubernetes Kubelet.
Jul 09 17:39:46.766159 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.766146    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 17:39:46.768180 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.768163    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 17:39:46.769698 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.768758    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-100.ec2.internal.18c0afd627c91d58  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-100.ec2.internal,UID:ip-10-0-136-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-100.ec2.internal,},FirstTimestamp:2026-07-09 17:39:46.764070232 +0000 UTC m=+0.366955679,LastTimestamp:2026-07-09 17:39:46.764070232 +0000 UTC m=+0.366955679,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-100.ec2.internal,}"
Jul 09 17:39:46.770106 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.770091    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjmnf"
Jul 09 17:39:46.770913 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.770891    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 17:39:46.772333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.772313    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 17:39:46.772333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.772330    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:46.772482 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.772424    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:46.773328 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773311    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 17:39:46.773328 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773313    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 17:39:46.773501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773337    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 17:39:46.773501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773435    2577 factory.go:55] Registering systemd factory
Jul 09 17:39:46.773501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773442    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 17:39:46.773501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773451    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 09 17:39:46.773501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773454    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 17:39:46.773753 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773711    2577 factory.go:153] Registering CRI-O factory
Jul 09 17:39:46.773753 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773728    2577 factory.go:223] Registration of the crio container factory successfully
Jul 09 17:39:46.773944 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773787    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 17:39:46.773944 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773813    2577 factory.go:103] Registering Raw factory
Jul 09 17:39:46.773944 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.773831    2577 manager.go:1196] Started watching for new ooms in manager
Jul 09 17:39:46.774231 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.774213    2577 manager.go:349] Starting recovery of all containers
Jul 09 17:39:46.777486 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.777463    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjmnf"
Jul 09 17:39:46.780379 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.780340    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 17:39:46.780479 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.780425    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 17:39:46.785607 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.785499    2577 manager.go:354] Recovery completed
Jul 09 17:39:46.789624 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.789610    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:46.794045 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794029    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:46.794133 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794061    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:46.794133 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794077    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:46.794686 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794669    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 17:39:46.794686 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794685    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 17:39:46.794791 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794700    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:46.794791 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.794703    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:46.795433 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795418    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:46.795501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795442    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:46.795501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795453    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:46.795823 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795808    2577 policy_none.go:49] "None policy: Start"
Jul 09 17:39:46.795884 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795829    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 17:39:46.795884 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.795845    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 17:39:46.796801 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.796733    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-100.ec2.internal.18c0afd629927f3e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-100.ec2.internal,UID:ip-10-0-136-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-100.ec2.internal,},FirstTimestamp:2026-07-09 17:39:46.794045246 +0000 UTC m=+0.396930701,LastTimestamp:2026-07-09 17:39:46.794045246 +0000 UTC m=+0.396930701,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-100.ec2.internal,}"
Jul 09 17:39:46.796955 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.796942    2577 policy_none.go:47] "Start"
Jul 09 17:39:46.835976 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.835962    2577 manager.go:337] "Starting Device Plugin manager"
Jul 09 17:39:46.836072 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.835997    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 17:39:46.836072 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836010    2577 server.go:87] "Starting device plugin registration server"
Jul 09 17:39:46.836267 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836254    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 17:39:46.836331 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836269    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 17:39:46.836436 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836420    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 17:39:46.836523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836503    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 17:39:46.836523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.836513    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 17:39:46.837009 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.836987    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 17:39:46.837073 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.837036    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:46.936891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.936854    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:46.938054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.937814    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:46.938054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.937842    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:46.938054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.937852    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:46.938054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.937878    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-100.ec2.internal"
Jul 09 17:39:46.945939 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.945924    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-100.ec2.internal"
Jul 09 17:39:46.945993 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.945946    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-100.ec2.internal\": node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:46.959294 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.959252    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 17:39:46.960451 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.960430    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 17:39:46.960536 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.960466    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 17:39:46.960536 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.960494    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 17:39:46.960536 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.960530    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 17:39:46.961037 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:46.961014    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:46.962161 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:46.962137    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:47.061166 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.061064    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"]
Jul 09 17:39:47.061166 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.061081    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.061396 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.061176    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:47.062154 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.062139    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:47.062245 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.062173    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:47.062245 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.062187    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:47.063582 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.063565    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:47.063707 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.063690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.063752 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.063726    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:47.065612 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065596    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:47.065612 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065608    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:47.065747 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065628    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:47.065747 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065643    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:47.065747 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065628    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:47.065747 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.065716    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:47.066813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.066796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.066892 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.066826    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:47.067494 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.067475    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:47.067565 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.067508    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:47.067565 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.067523    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:47.075041 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.075024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.075122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.075054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40a25d067cd687c255b64734b988a9bd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-100.ec2.internal\" (UID: \"40a25d067cd687c255b64734b988a9bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.075122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.075077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.091046 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.091022    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-100.ec2.internal\" not found" node="ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.095681 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.095665    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-100.ec2.internal\" not found" node="ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.161618 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.161580    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.175512 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40a25d067cd687c255b64734b988a9bd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-100.ec2.internal\" (UID: \"40a25d067cd687c255b64734b988a9bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.175683 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.175683 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.175683 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/40a25d067cd687c255b64734b988a9bd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-100.ec2.internal\" (UID: \"40a25d067cd687c255b64734b988a9bd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.175683 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.175683 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.175611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aa02da7a885957a80651181343104217-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal\" (UID: \"aa02da7a885957a80651181343104217\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.262225 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.262199    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.362881 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.362779    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.393351 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.393324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.399069 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.399052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:47.462959 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.462909    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.563463 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.563420    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.664025 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.663948    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.717449 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.717419    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 17:39:47.718008 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.717590    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:47.741632 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.741608    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:47.764666 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.764641    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.781487 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.781458    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:34:46 +0000 UTC" deadline="2027-12-20 09:06:17.056888605 +0000 UTC"
Jul 09 17:39:47.781487 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.781481    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12687h26m29.275410664s"
Jul 09 17:39:47.865447 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.865422    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:47.951068 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.951047    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:47.954661 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:47.954632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa02da7a885957a80651181343104217.slice/crio-bea900d5f190cc2656f0be27a836cd7497f296cee0b342e6269f8021f997fb73 WatchSource:0}: Error finding container bea900d5f190cc2656f0be27a836cd7497f296cee0b342e6269f8021f997fb73: Status 404 returned error can't find the container with id bea900d5f190cc2656f0be27a836cd7497f296cee0b342e6269f8021f997fb73
Jul 09 17:39:47.955088 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:47.955068    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a25d067cd687c255b64734b988a9bd.slice/crio-20da0e6b398f88cb8687efb9b7f2286301610252e457f324dafdd58bc4bab3a3 WatchSource:0}: Error finding container 20da0e6b398f88cb8687efb9b7f2286301610252e457f324dafdd58bc4bab3a3: Status 404 returned error can't find the container with id 20da0e6b398f88cb8687efb9b7f2286301610252e457f324dafdd58bc4bab3a3
Jul 09 17:39:47.961309 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.959911    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:39:47.963811 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.963766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal" event={"ID":"aa02da7a885957a80651181343104217","Type":"ContainerStarted","Data":"bea900d5f190cc2656f0be27a836cd7497f296cee0b342e6269f8021f997fb73"}
Jul 09 17:39:47.965166 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:47.965141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal" event={"ID":"40a25d067cd687c255b64734b988a9bd","Type":"ContainerStarted","Data":"20da0e6b398f88cb8687efb9b7f2286301610252e457f324dafdd58bc4bab3a3"}
Jul 09 17:39:47.966238 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:47.966221    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-100.ec2.internal\" not found"
Jul 09 17:39:48.021990 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.021965    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:48.072878 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.072857    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:48.085829 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.085812    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:48.086969 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.086956    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal"
Jul 09 17:39:48.106762 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.106745    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:48.753781 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.753752    2577 apiserver.go:52] "Watching apiserver"
Jul 09 17:39:48.763929 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.763900    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 17:39:48.764350 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.764318    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-llc92","openshift-network-operator/iptables-alerter-5bbhv","openshift-ovn-kubernetes/ovnkube-node-txgxp","kube-system/konnectivity-agent-xvfv6","kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8","openshift-dns/node-resolver-hg4dz","openshift-image-registry/node-ca-757rc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal","openshift-multus/multus-25qdq","openshift-multus/multus-additional-cni-plugins-27sfj","kube-system/global-pull-secret-syncer-ss8kv","openshift-cluster-node-tuning-operator/tuned-9d5x8","openshift-multus/network-metrics-daemon-dbfs7"]
Jul 09 17:39:48.766002 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.765979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.767488 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.767465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.768876 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.768804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.769081 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.769063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.769257 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.769238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5r8k5\""
Jul 09 17:39:48.769333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.769266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 17:39:48.770792 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.769572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.770792 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.769933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.770792 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.770084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.770792 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.770513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.770792 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.770513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87g9d\""
Jul 09 17:39:48.771439 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.771425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.772149 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.772122    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 17:39:48.772243 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.772188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5wk6\""
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.772699    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.772796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.772808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.772886    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.773066    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.773079    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.773083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.773221 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.773083    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774331    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kf7hl\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774504    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774627    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rmvhx\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 17:39:48.775137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.774814    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.778220 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.776762    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.779416 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.779351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhzg5\""
Jul 09 17:39:48.779619 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.779585    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.779704 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.779632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 17:39:48.779704 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.779692    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 17:39:48.779824 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.779805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.780020 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.780004    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 17:39:48.780098 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.780077    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.780153 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.780124    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.780201 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.780166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mts7w\""
Jul 09 17:39:48.780475 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.780456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.782193 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.782283 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.782240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-kubelet\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-conf\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-sys\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-var-lib-kubelet\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.782814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-tuned\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gc8\" (UniqueName: \"kubernetes.io/projected/105045a6-80b7-4c34-8cfd-03248d4e3b16-kube-api-access-p5gc8\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-sys-fs\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysconfig\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-netns\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.782996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-netd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-config\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-script-lib\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-modprobe-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-systemd\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7q8m\" (UniqueName: \"kubernetes.io/projected/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-kube-api-access-w7q8m\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 17:39:48.783182 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-slash\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783190    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-systemd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-log-socket\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d0e87c-b00d-40ec-81cc-95e594606af8-agent-certs\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a13b98e-d5f8-46db-8552-838b46760281-host\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-device-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-run\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783356    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-var-lib-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-etc-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-ovn\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-env-overrides\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d0e87c-b00d-40ec-81cc-95e594606af8-konnectivity-ca\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.783813 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.783695    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2v544\""
Jul 09 17:39:48.784714 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-registration-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.784784 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn2pd\" (UniqueName: \"kubernetes.io/projected/31125d0b-feac-4cb5-a5d5-435164782de8-kube-api-access-xn2pd\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.784784 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-kubernetes\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-lib-modules\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-bin\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-hosts-file\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4rt\" (UniqueName: \"kubernetes.io/projected/6a13b98e-d5f8-46db-8552-838b46760281-kube-api-access-bb4rt\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-socket-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-host\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovn-node-metrics-cert\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.784979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-tmp-dir\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a13b98e-d5f8-46db-8552-838b46760281-serviceca\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-tmp\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.785122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-systemd-units\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.785523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-node-log\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.785523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7l2r\" (UniqueName: \"kubernetes.io/projected/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-kube-api-access-w7l2r\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.785523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.785305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.786861 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.786817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:48.787070 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.787040    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:48.788355 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.788329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 17:39:48.788526 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.788494    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:48.788625 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.788569    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:48.788806 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.788771    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mlbtz\""
Jul 09 17:39:48.788806 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.788784    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:48.819486 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.819461    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6rhl8"
Jul 09 17:39:48.827295 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.827264    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6rhl8"
Jul 09 17:39:48.874587 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.874550    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 17:39:48.885680 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a13b98e-d5f8-46db-8552-838b46760281-host\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.885827 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-device-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.885827 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-env-overrides\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.885827 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-os-release\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.885827 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a13b98e-d5f8-46db-8552-838b46760281-host\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.885827 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-os-release\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-lib-modules\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-device-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-bin\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cni-binary-copy\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-hostroot\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-bin\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4rt\" (UniqueName: \"kubernetes.io/projected/6a13b98e-d5f8-46db-8552-838b46760281-kube-api-access-bb4rt\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-socket-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.885989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-lib-modules\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.886010 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-kubelet\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886332 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-env-overrides\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97r8\" (UniqueName: \"kubernetes.io/projected/30a61f7d-2778-48d8-b750-6570eccbf1b5-kube-api-access-z97r8\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:48.886578 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-systemd-units\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886578 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7l2r\" (UniqueName: \"kubernetes.io/projected/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-kube-api-access-w7l2r\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886578 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-socket-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.886578 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-k8s-cni-cncf-io\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-bin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvgq\" (UniqueName: \"kubernetes.io/projected/04cf20b8-27c4-48ec-bd17-69fa01f5256d-kube-api-access-slvgq\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-kubelet-config\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-kubelet\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-var-lib-kubelet\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7dt\" (UniqueName: \"kubernetes.io/projected/69b094cf-1239-4c18-9511-db2b7c29c5b8-kube-api-access-2g7dt\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cnibin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.886801 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-systemd-units\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-tuned\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-sys-fs\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-netns\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-var-lib-kubelet\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.886978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-netd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-script-lib\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-netns\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54dc526f-a5fd-40fb-9e91-2f5784348b86-host-slash\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-sys-fs\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-cni-netd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887134    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-netns\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-slash\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-kubelet\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d0e87c-b00d-40ec-81cc-95e594606af8-agent-certs\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-slash\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-run\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-run\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-var-lib-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-var-lib-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-etc-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-etc-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-ovn\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-ovn\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d0e87c-b00d-40ec-81cc-95e594606af8-konnectivity-ca\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.887795 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-hosts-file\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-system-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-socket-dir-parent\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-script-lib\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-registration-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn2pd\" (UniqueName: \"kubernetes.io/projected/31125d0b-feac-4cb5-a5d5-435164782de8-kube-api-access-xn2pd\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-kubernetes\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-hosts-file\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-conf-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-cnibin\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-kubernetes\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-host\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovn-node-metrics-cert\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.888678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-tmp-dir\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-etc-kubernetes\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.887769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-registration-dir\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-dbus\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888059    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a13b98e-d5f8-46db-8552-838b46760281-serviceca\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-tmp\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/13d0e87c-b00d-40ec-81cc-95e594606af8-konnectivity-ca\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-node-log\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-conf\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-sys\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-host\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.889496 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-tmp-dir\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-run-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-node-log\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-multus\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gc8\" (UniqueName: \"kubernetes.io/projected/105045a6-80b7-4c34-8cfd-03248d4e3b16-kube-api-access-p5gc8\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31125d0b-feac-4cb5-a5d5-435164782de8-etc-selinux\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysconfig\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysctl-conf\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6a13b98e-d5f8-46db-8552-838b46760281-serviceca\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-sys\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-sysconfig\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-config\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-multus-certs\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-modprobe-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-systemd\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.890334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-modprobe-d\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7q8m\" (UniqueName: \"kubernetes.io/projected/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-kube-api-access-w7q8m\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54dc526f-a5fd-40fb-9e91-2f5784348b86-iptables-alerter-script\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.888998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-systemd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-log-socket\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-daemon-config\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8v4p\" (UniqueName: \"kubernetes.io/projected/54dc526f-a5fd-40fb-9e91-2f5784348b86-kube-api-access-w8v4p\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-systemd\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovnkube-config\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-systemd\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-run-openvswitch\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.891142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.889231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-log-socket\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.892004 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.891961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-tmp\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.892119 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.892095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-ovn-node-metrics-cert\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.892407 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.892384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/105045a6-80b7-4c34-8cfd-03248d4e3b16-etc-tuned\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.892542 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.892523    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/13d0e87c-b00d-40ec-81cc-95e594606af8-agent-certs\") pod \"konnectivity-agent-xvfv6\" (UID: \"13d0e87c-b00d-40ec-81cc-95e594606af8\") " pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:48.898579 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.898522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4rt\" (UniqueName: \"kubernetes.io/projected/6a13b98e-d5f8-46db-8552-838b46760281-kube-api-access-bb4rt\") pod \"node-ca-757rc\" (UID: \"6a13b98e-d5f8-46db-8552-838b46760281\") " pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:48.899152 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.899126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7l2r\" (UniqueName: \"kubernetes.io/projected/c45a8f89-5a42-4d46-9d90-3f0718b5f72b-kube-api-access-w7l2r\") pod \"ovnkube-node-txgxp\" (UID: \"c45a8f89-5a42-4d46-9d90-3f0718b5f72b\") " pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:48.901222 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.901200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gc8\" (UniqueName: \"kubernetes.io/projected/105045a6-80b7-4c34-8cfd-03248d4e3b16-kube-api-access-p5gc8\") pod \"tuned-9d5x8\" (UID: \"105045a6-80b7-4c34-8cfd-03248d4e3b16\") " pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:48.901319 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.901270    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:48.901319 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.901291    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:48.901319 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.901310    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:48.901495 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.901439    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.401351813 +0000 UTC m=+3.004237248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:48.903895 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.903841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7q8m\" (UniqueName: \"kubernetes.io/projected/2d7d8c3a-52bd-4089-8af7-90a74d3abfab-kube-api-access-w7q8m\") pod \"node-resolver-hg4dz\" (UID: \"2d7d8c3a-52bd-4089-8af7-90a74d3abfab\") " pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:48.904651 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.904629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn2pd\" (UniqueName: \"kubernetes.io/projected/31125d0b-feac-4cb5-a5d5-435164782de8-kube-api-access-xn2pd\") pod \"aws-ebs-csi-driver-node-zv2q8\" (UID: \"31125d0b-feac-4cb5-a5d5-435164782de8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:48.990298 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cni-binary-copy\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990298 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-hostroot\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-kubelet\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97r8\" (UniqueName: \"kubernetes.io/projected/30a61f7d-2778-48d8-b750-6570eccbf1b5-kube-api-access-z97r8\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-k8s-cni-cncf-io\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-hostroot\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-kubelet\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-bin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-k8s-cni-cncf-io\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvgq\" (UniqueName: \"kubernetes.io/projected/04cf20b8-27c4-48ec-bd17-69fa01f5256d-kube-api-access-slvgq\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-bin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-kubelet-config\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7dt\" (UniqueName: \"kubernetes.io/projected/69b094cf-1239-4c18-9511-db2b7c29c5b8-kube-api-access-2g7dt\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.990563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cnibin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990599    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-netns\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54dc526f-a5fd-40fb-9e91-2f5784348b86-host-slash\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-system-cni-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cnibin\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/54dc526f-a5fd-40fb-9e91-2f5784348b86-host-slash\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-system-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-system-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-socket-dir-parent\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-kubelet-config\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-conf-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-netns\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-conf-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-socket-dir-parent\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-cnibin\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-cni-binary-copy\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990975    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-cnibin\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-etc-kubernetes\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.990996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-etc-kubernetes\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-dbus\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.991057    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.991129    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.491111808 +0000 UTC m=+3.093997242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-multus\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-cni-dir\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-multus-certs\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-run-multus-certs\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-dbus\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.991988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54dc526f-a5fd-40fb-9e91-2f5784348b86-iptables-alerter-script\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-daemon-config\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-host-var-lib-cni-multus\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8v4p\" (UniqueName: \"kubernetes.io/projected/54dc526f-a5fd-40fb-9e91-2f5784348b86-kube-api-access-w8v4p\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-os-release\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.991613    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-os-release\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:48.991672    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:39:49.491656949 +0000 UTC m=+3.094542383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/69b094cf-1239-4c18-9511-db2b7c29c5b8-os-release\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04cf20b8-27c4-48ec-bd17-69fa01f5256d-os-release\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/69b094cf-1239-4c18-9511-db2b7c29c5b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04cf20b8-27c4-48ec-bd17-69fa01f5256d-multus-daemon-config\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:48.992621 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:48.991836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/54dc526f-a5fd-40fb-9e91-2f5784348b86-iptables-alerter-script\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:49.004498 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.004438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvgq\" (UniqueName: \"kubernetes.io/projected/04cf20b8-27c4-48ec-bd17-69fa01f5256d-kube-api-access-slvgq\") pod \"multus-25qdq\" (UID: \"04cf20b8-27c4-48ec-bd17-69fa01f5256d\") " pod="openshift-multus/multus-25qdq"
Jul 09 17:39:49.004644 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.004627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97r8\" (UniqueName: \"kubernetes.io/projected/30a61f7d-2778-48d8-b750-6570eccbf1b5-kube-api-access-z97r8\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:49.008614 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.008594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8v4p\" (UniqueName: \"kubernetes.io/projected/54dc526f-a5fd-40fb-9e91-2f5784348b86-kube-api-access-w8v4p\") pod \"iptables-alerter-5bbhv\" (UID: \"54dc526f-a5fd-40fb-9e91-2f5784348b86\") " pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:49.012290 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.012271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7dt\" (UniqueName: \"kubernetes.io/projected/69b094cf-1239-4c18-9511-db2b7c29c5b8-kube-api-access-2g7dt\") pod \"multus-additional-cni-plugins-27sfj\" (UID: \"69b094cf-1239-4c18-9511-db2b7c29c5b8\") " pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:49.080272 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.080235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8"
Jul 09 17:39:49.087597 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.087572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9d5x8"
Jul 09 17:39:49.096502 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.096476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:39:49.102158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.102139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg4dz"
Jul 09 17:39:49.110832 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.110811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:39:49.118314 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.118295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-757rc"
Jul 09 17:39:49.125815 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.125794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25qdq"
Jul 09 17:39:49.133349 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.133332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-27sfj"
Jul 09 17:39:49.139908 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.139890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5bbhv"
Jul 09 17:39:49.142706 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.142686    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:49.494534 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.494446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:49.494534 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.494502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.494559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494620    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494643    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494647    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494657    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494650    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.494694228 +0000 UTC m=+4.097579668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.494722319 +0000 UTC m=+4.097607758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:49.494761 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.494750    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.494740897 +0000 UTC m=+4.097626335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:49.770836 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.770812    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d0e87c_b00d_40ec_81cc_95e594606af8.slice/crio-fe93c0b73217e8b6c273130bda276eddc0498bf6fa7b31a73de97a063940d953 WatchSource:0}: Error finding container fe93c0b73217e8b6c273130bda276eddc0498bf6fa7b31a73de97a063940d953: Status 404 returned error can't find the container with id fe93c0b73217e8b6c273130bda276eddc0498bf6fa7b31a73de97a063940d953
Jul 09 17:39:49.772149 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.772124    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cf20b8_27c4_48ec_bd17_69fa01f5256d.slice/crio-734107cf82e3ad9c2aa6cddbb0d57fe583293f957de1e31e69d0163293852cac WatchSource:0}: Error finding container 734107cf82e3ad9c2aa6cddbb0d57fe583293f957de1e31e69d0163293852cac: Status 404 returned error can't find the container with id 734107cf82e3ad9c2aa6cddbb0d57fe583293f957de1e31e69d0163293852cac
Jul 09 17:39:49.775215 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.775152    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7d8c3a_52bd_4089_8af7_90a74d3abfab.slice/crio-7de244e90b839ebf79299d02a9769eb0ee58f23f88f6c13ec020ec88a365e8f1 WatchSource:0}: Error finding container 7de244e90b839ebf79299d02a9769eb0ee58f23f88f6c13ec020ec88a365e8f1: Status 404 returned error can't find the container with id 7de244e90b839ebf79299d02a9769eb0ee58f23f88f6c13ec020ec88a365e8f1
Jul 09 17:39:49.776506 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.776487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31125d0b_feac_4cb5_a5d5_435164782de8.slice/crio-29774f9d6d63e258c55e6fd58d58763142a3a6752f81ea762002e3bc26a0707c WatchSource:0}: Error finding container 29774f9d6d63e258c55e6fd58d58763142a3a6752f81ea762002e3bc26a0707c: Status 404 returned error can't find the container with id 29774f9d6d63e258c55e6fd58d58763142a3a6752f81ea762002e3bc26a0707c
Jul 09 17:39:49.777806 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.777781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b094cf_1239_4c18_9511_db2b7c29c5b8.slice/crio-266fe99a8cad8a57ad5de6a128fa279a58af9eb7df2262af2462ee68d1344ae8 WatchSource:0}: Error finding container 266fe99a8cad8a57ad5de6a128fa279a58af9eb7df2262af2462ee68d1344ae8: Status 404 returned error can't find the container with id 266fe99a8cad8a57ad5de6a128fa279a58af9eb7df2262af2462ee68d1344ae8
Jul 09 17:39:49.778637 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.778599    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45a8f89_5a42_4d46_9d90_3f0718b5f72b.slice/crio-8a6e07351f74e27fcf9093ff036924656db2fca7c2b86491eda03cf53ae0ea0a WatchSource:0}: Error finding container 8a6e07351f74e27fcf9093ff036924656db2fca7c2b86491eda03cf53ae0ea0a: Status 404 returned error can't find the container with id 8a6e07351f74e27fcf9093ff036924656db2fca7c2b86491eda03cf53ae0ea0a
Jul 09 17:39:49.780193 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.780138    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105045a6_80b7_4c34_8cfd_03248d4e3b16.slice/crio-70496e316848c7fac394315b4c6d4b6398d823ce83dccb457f9fca948e954e95 WatchSource:0}: Error finding container 70496e316848c7fac394315b4c6d4b6398d823ce83dccb457f9fca948e954e95: Status 404 returned error can't find the container with id 70496e316848c7fac394315b4c6d4b6398d823ce83dccb457f9fca948e954e95
Jul 09 17:39:49.781536 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.781511    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a13b98e_d5f8_46db_8552_838b46760281.slice/crio-d4f0a702a1ddfcbe1e18eb439cdad2c26b6c61475a90a973a9ea5390550487a9 WatchSource:0}: Error finding container d4f0a702a1ddfcbe1e18eb439cdad2c26b6c61475a90a973a9ea5390550487a9: Status 404 returned error can't find the container with id d4f0a702a1ddfcbe1e18eb439cdad2c26b6c61475a90a973a9ea5390550487a9
Jul 09 17:39:49.782546 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:39:49.782522    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dc526f_a5fd_40fb_9e91_2f5784348b86.slice/crio-ec7af84dd293976415db3ce2a7733bbca1b6646e84a5986418de58b0636a96be WatchSource:0}: Error finding container ec7af84dd293976415db3ce2a7733bbca1b6646e84a5986418de58b0636a96be: Status 404 returned error can't find the container with id ec7af84dd293976415db3ce2a7733bbca1b6646e84a5986418de58b0636a96be
Jul 09 17:39:49.829689 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.829531    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:48 +0000 UTC" deadline="2028-01-01 02:50:19.665304089 +0000 UTC"
Jul 09 17:39:49.829689 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.829689    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12969h10m29.835618937s"
Jul 09 17:39:49.961090 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.961061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:49.961234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.961061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:49.961234 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.961213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:49.961350 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:49.961275    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:49.968487 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.968464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bbhv" event={"ID":"54dc526f-a5fd-40fb-9e91-2f5784348b86","Type":"ContainerStarted","Data":"ec7af84dd293976415db3ce2a7733bbca1b6646e84a5986418de58b0636a96be"}
Jul 09 17:39:49.969479 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.969441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-757rc" event={"ID":"6a13b98e-d5f8-46db-8552-838b46760281","Type":"ContainerStarted","Data":"d4f0a702a1ddfcbe1e18eb439cdad2c26b6c61475a90a973a9ea5390550487a9"}
Jul 09 17:39:49.970632 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.970610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"8a6e07351f74e27fcf9093ff036924656db2fca7c2b86491eda03cf53ae0ea0a"}
Jul 09 17:39:49.971674 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.971649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerStarted","Data":"266fe99a8cad8a57ad5de6a128fa279a58af9eb7df2262af2462ee68d1344ae8"}
Jul 09 17:39:49.972678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.972652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8" event={"ID":"31125d0b-feac-4cb5-a5d5-435164782de8","Type":"ContainerStarted","Data":"29774f9d6d63e258c55e6fd58d58763142a3a6752f81ea762002e3bc26a0707c"}
Jul 09 17:39:49.973716 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.973695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg4dz" event={"ID":"2d7d8c3a-52bd-4089-8af7-90a74d3abfab","Type":"ContainerStarted","Data":"7de244e90b839ebf79299d02a9769eb0ee58f23f88f6c13ec020ec88a365e8f1"}
Jul 09 17:39:49.974712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.974692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25qdq" event={"ID":"04cf20b8-27c4-48ec-bd17-69fa01f5256d","Type":"ContainerStarted","Data":"734107cf82e3ad9c2aa6cddbb0d57fe583293f957de1e31e69d0163293852cac"}
Jul 09 17:39:49.976351 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.976325    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal" event={"ID":"40a25d067cd687c255b64734b988a9bd","Type":"ContainerStarted","Data":"e896e029368d7d6f07739256e14974de8e45a9a08fefc47ebb043a03ed90c957"}
Jul 09 17:39:49.977392 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.977355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9d5x8" event={"ID":"105045a6-80b7-4c34-8cfd-03248d4e3b16","Type":"ContainerStarted","Data":"70496e316848c7fac394315b4c6d4b6398d823ce83dccb457f9fca948e954e95"}
Jul 09 17:39:49.978292 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.978235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvfv6" event={"ID":"13d0e87c-b00d-40ec-81cc-95e594606af8","Type":"ContainerStarted","Data":"fe93c0b73217e8b6c273130bda276eddc0498bf6fa7b31a73de97a063940d953"}
Jul 09 17:39:49.993887 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:49.993847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-100.ec2.internal" podStartSLOduration=1.993833615 podStartE2EDuration="1.993833615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:49.993582163 +0000 UTC m=+3.596467620" watchObservedRunningTime="2026-07-09 17:39:49.993833615 +0000 UTC m=+3.596719070"
Jul 09 17:39:50.501247 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.501157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:50.501247 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.501223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:50.501553 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.501278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:50.501553 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.501474    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:50.501553 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.501537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:52.501519843 +0000 UTC m=+6.104405290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:50.502022 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.501975    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:50.502022 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.502001    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:50.502022 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.502015    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:50.502209 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.502062    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:52.502047876 +0000 UTC m=+6.104933324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:50.502209 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.502127    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:50.502209 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.502157    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:39:52.502147295 +0000 UTC m=+6.105032729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:50.830762 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.830716    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:48 +0000 UTC" deadline="2028-04-23 22:52:24.072149458 +0000 UTC"
Jul 09 17:39:50.830762 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.830761    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15701h12m33.241393367s"
Jul 09 17:39:50.963840 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:50.963808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:50.964027 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:50.963937    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:51.008617 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:51.008578    2577 generic.go:356] "Generic (PLEG): container finished" podID="aa02da7a885957a80651181343104217" containerID="2d47ed60311f9ee870775b6d065c4baee4d572115904c3c01cce194e2444916e" exitCode=0
Jul 09 17:39:51.009500 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:51.009472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal" event={"ID":"aa02da7a885957a80651181343104217","Type":"ContainerDied","Data":"2d47ed60311f9ee870775b6d065c4baee4d572115904c3c01cce194e2444916e"}
Jul 09 17:39:51.961640 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:51.961600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:51.962126 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:51.961761    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:51.962126 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:51.961858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:51.962126 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:51.961933    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:52.030211 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.030168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal" event={"ID":"aa02da7a885957a80651181343104217","Type":"ContainerStarted","Data":"65c26159bcab7bf331ebf0f8b5a677672121426afa11aff868838eac7f8d9e46"}
Jul 09 17:39:52.050956 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.050898    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-100.ec2.internal" podStartSLOduration=4.050876747 podStartE2EDuration="4.050876747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:52.050874009 +0000 UTC m=+5.653759465" watchObservedRunningTime="2026-07-09 17:39:52.050876747 +0000 UTC m=+5.653762204"
Jul 09 17:39:52.520270 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.520229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:52.520484 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.520295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:52.520484 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.520372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:52.520605 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.520518    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:52.520605 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.520584    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:39:56.520564784 +0000 UTC m=+10.123450231 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:52.520993 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.520971    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:52.521075 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.521032    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:56.521014451 +0000 UTC m=+10.123899906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:52.521136 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.521120    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:52.521191 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.521139    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:52.521191 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.521152    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:52.521191 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.521189    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:39:56.52117714 +0000 UTC m=+10.124062591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:52.961151 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:52.961120    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:52.961328 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:52.961259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:53.961089 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:53.961042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:53.961571 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:53.961178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:53.961571 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:53.961242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:53.961571 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:53.961322    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:54.964932 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:54.964845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:54.965529 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:54.964984    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:55.961158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:55.961096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:55.961333 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:55.961241    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:55.961665 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:55.961095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:55.961665 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:55.961621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:56.557848 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:56.557807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:56.557868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:56.557909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.557971    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558010    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558067    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558081    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.55802618 +0000 UTC m=+18.160911623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558096    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558104    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.558093032 +0000 UTC m=+18.160978466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558112    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:56.558338 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.558158    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.558141983 +0000 UTC m=+18.161027419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:56.964622 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:56.964586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:56.964784 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:56.964732    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:57.960862 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:57.960831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:57.960862 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:57.960859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:57.961412 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:57.960979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:57.961412 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:57.961080    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:39:58.964295 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:58.964268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:39:58.964745 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:58.964397    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:39:59.961083 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:59.961038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:39:59.961267 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:39:59.961091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:39:59.961267 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:59.961186    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:39:59.961395 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:39:59.961299    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:00.960997 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:00.960961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:00.961481 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:00.961092    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:01.961347 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:01.961299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:01.961347 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:01.961343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:01.961837 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:01.961480    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:01.961837 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:01.961624    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:02.961333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:02.961302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:02.961516 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:02.961438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:03.961100 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:03.961070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:03.961335 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:03.961190    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:03.961335 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:03.961222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:03.961335 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:03.961286    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:04.624389 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:04.624332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:04.624415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:04.624467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624540    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624567    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624568    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624582    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624568    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:40:20.624623471 +0000 UTC m=+34.227508909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:20.624651464 +0000 UTC m=+34.227536902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:04.624885 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.624680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:40:20.624671087 +0000 UTC m=+34.227556521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:04.961688 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:04.961649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:04.961930 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:04.961802    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:05.960757 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:05.960698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:05.960757 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:05.960752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:05.961293 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:05.960852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:05.961293 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:05.960991    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:06.961939 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:06.961902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:06.962406 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:06.962013    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:07.961513 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:07.961480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:07.961712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:07.961486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:07.961712 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:07.961615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:07.961844 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:07.961718    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:08.961202 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:08.961176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:08.961829 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:08.961292    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:09.069922 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.069778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-757rc" event={"ID":"6a13b98e-d5f8-46db-8552-838b46760281","Type":"ContainerStarted","Data":"45d7df48c3df29723c43c8b15d95684e1b5696442c6ece1d96fc8cd0cbfed653"}
Jul 09 17:40:09.072353 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.072318    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:40:09.072891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.072862    2577 generic.go:356] "Generic (PLEG): container finished" podID="c45a8f89-5a42-4d46-9d90-3f0718b5f72b" containerID="4f1c501fb6c6dde92fbed12d0cadb20b6795913ebd35cce214c59652e6067121" exitCode=1
Jul 09 17:40:09.073004 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.072946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"aa3a497510fe71f2f25cab69d00afbbe3779578b66af3149b9abbebefd6353d2"}
Jul 09 17:40:09.073067 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.073029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"46b4bc8daf7795500b78fcd7bfeb0fc267f85889586205dbfe312716735fe698"}
Jul 09 17:40:09.073122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.073083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerDied","Data":"4f1c501fb6c6dde92fbed12d0cadb20b6795913ebd35cce214c59652e6067121"}
Jul 09 17:40:09.073175 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.073136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"44c2ea0fe4b9bb5d120c04ce941dbf0cf96311d1a19f3842d8b089cf1fee8259"}
Jul 09 17:40:09.074951 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.074925    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="b00e8432c3721c67c2ac9e7363066df890208766b45cbc3e382e0dd0663efd42" exitCode=0
Jul 09 17:40:09.075044 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.075004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"b00e8432c3721c67c2ac9e7363066df890208766b45cbc3e382e0dd0663efd42"}
Jul 09 17:40:09.076699 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.076563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8" event={"ID":"31125d0b-feac-4cb5-a5d5-435164782de8","Type":"ContainerStarted","Data":"06a2d9150ba35f4e9b940306888ce9b57259755279b53777fba4868bf7f16535"}
Jul 09 17:40:09.078151 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.078095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg4dz" event={"ID":"2d7d8c3a-52bd-4089-8af7-90a74d3abfab","Type":"ContainerStarted","Data":"7a828d5ec71a2a18289a96eaf5e4abd6d61f88c61c227802ec1c8e87dcf02a18"}
Jul 09 17:40:09.079794 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.079774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25qdq" event={"ID":"04cf20b8-27c4-48ec-bd17-69fa01f5256d","Type":"ContainerStarted","Data":"6b73ae101a9bbfa7b60e929c174cddefa5e3093607605696eebfe549d274fb90"}
Jul 09 17:40:09.081541 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.081476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9d5x8" event={"ID":"105045a6-80b7-4c34-8cfd-03248d4e3b16","Type":"ContainerStarted","Data":"b15311c75c6b19b92b2f44df848a425f86cbf590f55343643f19331023e490f9"}
Jul 09 17:40:09.083066 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.083032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xvfv6" event={"ID":"13d0e87c-b00d-40ec-81cc-95e594606af8","Type":"ContainerStarted","Data":"62ec16cee2444d535b0e825b595aae7bb867fa55f6c3c44607306f0bcc7891e8"}
Jul 09 17:40:09.088408 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.087420    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-757rc" podStartSLOduration=3.511382766 podStartE2EDuration="22.087395997s" totalImagesPullingTime="18.576013231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.087012272 +0000 UTC m=+22.689897728" watchObservedRunningTime="2026-07-09 17:40:09.087395997 +0000 UTC m=+22.690281453"
Jul 09 17:40:09.114636 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.114585    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hg4dz" podStartSLOduration=4.527761878 podStartE2EDuration="23.114570971s" totalImagesPullingTime="18.586809093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.113536935 +0000 UTC m=+22.716422391" watchObservedRunningTime="2026-07-09 17:40:09.114570971 +0000 UTC m=+22.717456426"
Jul 09 17:40:09.141713 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.141658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xvfv6" podStartSLOduration=11.757993123 podStartE2EDuration="22.141638661s" totalImagesPullingTime="10.383645538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.140605878 +0000 UTC m=+22.743491333" watchObservedRunningTime="2026-07-09 17:40:09.141638661 +0000 UTC m=+22.744524107"
Jul 09 17:40:09.177120 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.176878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-25qdq" podStartSLOduration=3.412119838 podStartE2EDuration="22.176860288s" totalImagesPullingTime="18.76474045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.17658181 +0000 UTC m=+22.779467267" watchObservedRunningTime="2026-07-09 17:40:09.176860288 +0000 UTC m=+22.779745747"
Jul 09 17:40:09.248921 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.248870    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9d5x8" podStartSLOduration=4.492609688 podStartE2EDuration="23.248838724s" totalImagesPullingTime="18.756229036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.247814374 +0000 UTC m=+22.850699829" watchObservedRunningTime="2026-07-09 17:40:09.248838724 +0000 UTC m=+22.851724180"
Jul 09 17:40:09.961329 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.961291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:09.961860 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:09.961434    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:09.961860 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.961503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:09.961860 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:09.961616    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:09.997490 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:09.997463    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 17:40:10.087777 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.087695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8" event={"ID":"31125d0b-feac-4cb5-a5d5-435164782de8","Type":"ContainerStarted","Data":"c4a38e9808532e11644bb67e9e0157b0ceb38f3f851eb290a2d93b4b8f7229e3"}
Jul 09 17:40:10.089306 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.089274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5bbhv" event={"ID":"54dc526f-a5fd-40fb-9e91-2f5784348b86","Type":"ContainerStarted","Data":"462ad9fd6242c5fc528eaf2c546e9df465f3f5e8e99a44de9d23a1f6e47ed959"}
Jul 09 17:40:10.092356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.092336    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:40:10.092755 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.092731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"c1f455f1cca93148f09e000eb37259e47ff372885c77e4b8c291013a7b23d9be"}
Jul 09 17:40:10.092853 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.092801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"8c6a5caeb0773776a7aa850f5a0f731f1e0cc2c55159736f9c57c80e717d04b4"}
Jul 09 17:40:10.113883 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.113764    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5bbhv" podStartSLOduration=4.363377032 podStartE2EDuration="23.113738129s" totalImagesPullingTime="18.750361097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:10.113687814 +0000 UTC m=+23.716573273" watchObservedRunningTime="2026-07-09 17:40:10.113738129 +0000 UTC m=+23.716623586"
Jul 09 17:40:10.848989 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.848878    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T17:40:09.997484438Z","UUID":"d77bde81-1897-4964-9357-d7ca3c31d3f1","Handler":null,"Name":"","Endpoint":""}
Jul 09 17:40:10.850524 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.850500    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 17:40:10.850652 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.850547    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 17:40:10.961316 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:10.961284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:10.961519 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:10.961448    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:11.098427 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:11.098386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8" event={"ID":"31125d0b-feac-4cb5-a5d5-435164782de8","Type":"ContainerStarted","Data":"7761e0e1cbdd3a7375ec6ef377bcd41f0151cbf93198866aabf19111504fdfa6"}
Jul 09 17:40:11.961461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:11.961430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:11.961640 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:11.961511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:11.961640 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:11.961572    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:11.961640 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:11.961609    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:12.104188 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:12.104158    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:40:12.104752 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:12.104711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"adecb58156a8d41b26aab93b02d53159818416dec4391048f014a577531f6a06"}
Jul 09 17:40:12.127406 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:12.127328    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zv2q8" podStartSLOduration=5.418366285 podStartE2EDuration="26.127310076s" totalImagesPullingTime="20.708943791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:12.127219107 +0000 UTC m=+25.730104589" watchObservedRunningTime="2026-07-09 17:40:12.127310076 +0000 UTC m=+25.730195535"
Jul 09 17:40:12.964118 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:12.963925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:12.964563 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:12.964194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:13.734278 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:13.733988    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:40:13.734499 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:13.734480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:40:13.961022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:13.960988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:13.961193 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:13.961095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:13.961193 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:13.961151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:13.961288 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:13.961248    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:14.111558 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.111487    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:40:14.112238 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.111881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"5fc3a69618beebedb2a3aa3b17e4ba8c079a970cb39ba42a3ddf8770ab29cda8"}
Jul 09 17:40:14.112238 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.112137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:14.112417 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.112332    2577 scope.go:117] "RemoveContainer" containerID="4f1c501fb6c6dde92fbed12d0cadb20b6795913ebd35cce214c59652e6067121"
Jul 09 17:40:14.113852 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.113828    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="2628a0a0f77ec9e7cd844857de31de2875d50529d7e9504a213b16a7e667aeb0" exitCode=0
Jul 09 17:40:14.113963 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.113911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"2628a0a0f77ec9e7cd844857de31de2875d50529d7e9504a213b16a7e667aeb0"}
Jul 09 17:40:14.114147 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.114115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:40:14.114558 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.114542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xvfv6"
Jul 09 17:40:14.129209 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.129188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:14.964644 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:14.964610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:14.964888 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:14.964729    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:15.119246 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.119220    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:40:15.119778 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.119737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" event={"ID":"c45a8f89-5a42-4d46-9d90-3f0718b5f72b","Type":"ContainerStarted","Data":"631d17c485443b878ffda6d1b201c1b0dbdc5f0982150ca5c3af87ffad887664"}
Jul 09 17:40:15.120077 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.120049    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:15.120187 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.120130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:15.136137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.136109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:15.156407 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.156064    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp" podStartSLOduration=10.271646147 podStartE2EDuration="29.156048383s" totalImagesPullingTime="18.884402236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:15.154400028 +0000 UTC m=+28.757285486" watchObservedRunningTime="2026-07-09 17:40:15.156048383 +0000 UTC m=+28.758933863"
Jul 09 17:40:15.524693 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.524622    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ss8kv"]
Jul 09 17:40:15.524853 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.524734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:15.524853 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:15.524815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:15.527894 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.527860    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbfs7"]
Jul 09 17:40:15.528009 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.527994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:15.528140 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:15.528110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:15.528639 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.528615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llc92"]
Jul 09 17:40:15.528732 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:15.528687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:15.528793 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:15.528749    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:16.123612 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:16.123577    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="55167af429b43fdc1e02885c30e1685cfea93acc63ca1327ccd9066d93d1c272" exitCode=0
Jul 09 17:40:16.124086 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:16.123651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"55167af429b43fdc1e02885c30e1685cfea93acc63ca1327ccd9066d93d1c272"}
Jul 09 17:40:16.962500 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:16.962308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:16.962687 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:16.962582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:17.961256 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:17.961225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:17.961762 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:17.961239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:17.961762 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:17.961352    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:17.961762 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:17.961454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:18.130882 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:18.130852    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="b6212924a5686eccb3f4e81db670f824b5898408db832081d98f320e1cfd237c" exitCode=0
Jul 09 17:40:18.130882 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:18.130888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"b6212924a5686eccb3f4e81db670f824b5898408db832081d98f320e1cfd237c"}
Jul 09 17:40:18.961311 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:18.961280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:18.961897 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:18.961414    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:19.961167 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:19.961135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:19.961336 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:19.961135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:19.961336 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:19.961283    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ss8kv" podUID="76cd4c1f-fee0-4f8c-b9ac-026346a096bb"
Jul 09 17:40:19.961820 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:19.961372    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbfs7" podUID="30a61f7d-2778-48d8-b750-6570eccbf1b5"
Jul 09 17:40:20.644066 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:20.644022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:20.644076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:20.644115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644228    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644234    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644251    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:40:20.644277 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644273    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:40:20.644562 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644287    2577 projected.go:196] Error preparing data for projected volume kube-api-access-wq58v for pod openshift-network-diagnostics/network-check-target-llc92: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:20.644562 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644290    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret podName:76cd4c1f-fee0-4f8c-b9ac-026346a096bb nodeName:}" failed. No retries permitted until 2026-07-09 17:40:52.644271125 +0000 UTC m=+66.247156561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret") pod "global-pull-secret-syncer-ss8kv" (UID: "76cd4c1f-fee0-4f8c-b9ac-026346a096bb") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:20.644562 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644318    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:52.64430131 +0000 UTC m=+66.247186758 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:20.644562 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.644340    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v podName:018d06aa-ce76-4485-95e0-e46b98c1f69c nodeName:}" failed. No retries permitted until 2026-07-09 17:40:52.644329614 +0000 UTC m=+66.247215050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wq58v" (UniqueName: "kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v") pod "network-check-target-llc92" (UID: "018d06aa-ce76-4485-95e0-e46b98c1f69c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:20.960923 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:20.960891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:20.961106 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:20.961008    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llc92" podUID="018d06aa-ce76-4485-95e0-e46b98c1f69c"
Jul 09 17:40:21.219251 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.219161    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-100.ec2.internal" event="NodeReady"
Jul 09 17:40:21.219811 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.219313    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 17:40:21.281924 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.281890    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vc6k"]
Jul 09 17:40:21.308270 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.308239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5hr5d"]
Jul 09 17:40:21.308451 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.308387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.312321 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.312300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 17:40:21.312695 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.312674    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lljhl\""
Jul 09 17:40:21.312794 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.312731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 17:40:21.334323 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.334296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vc6k"]
Jul 09 17:40:21.334471 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.334350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hr5d"]
Jul 09 17:40:21.334471 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.334457    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.344006 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.343978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 17:40:21.344297 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.344277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 17:40:21.347725 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.344838    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nnsbc\""
Jul 09 17:40:21.347725 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.345794    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 17:40:21.451313 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34e57e5-2188-471a-91ff-92b558e41ea6-tmp-dir\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.451504 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ckvf\" (UniqueName: \"kubernetes.io/projected/f34e57e5-2188-471a-91ff-92b558e41ea6-kube-api-access-9ckvf\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.451504 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7d8c\" (UniqueName: \"kubernetes.io/projected/c910f899-63b6-466d-adaa-b66ed83517fa-kube-api-access-l7d8c\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.451504 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.451676 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c910f899-63b6-466d-adaa-b66ed83517fa-cert\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.451676 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.451552    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f34e57e5-2188-471a-91ff-92b558e41ea6-config-volume\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.552613 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7d8c\" (UniqueName: \"kubernetes.io/projected/c910f899-63b6-466d-adaa-b66ed83517fa-kube-api-access-l7d8c\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.552613 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.552818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c910f899-63b6-466d-adaa-b66ed83517fa-cert\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.552818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f34e57e5-2188-471a-91ff-92b558e41ea6-config-volume\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.552818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34e57e5-2188-471a-91ff-92b558e41ea6-tmp-dir\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.552818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.552734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ckvf\" (UniqueName: \"kubernetes.io/projected/f34e57e5-2188-471a-91ff-92b558e41ea6-kube-api-access-9ckvf\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.552818 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:21.552780    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:21.553040 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:21.552852    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:22.052831354 +0000 UTC m=+35.655716793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:21.553124 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.553101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34e57e5-2188-471a-91ff-92b558e41ea6-tmp-dir\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.553470 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.553441    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f34e57e5-2188-471a-91ff-92b558e41ea6-config-volume\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.557520 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.557501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c910f899-63b6-466d-adaa-b66ed83517fa-cert\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.564198 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.564170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7d8c\" (UniqueName: \"kubernetes.io/projected/c910f899-63b6-466d-adaa-b66ed83517fa-kube-api-access-l7d8c\") pod \"ingress-canary-5hr5d\" (UID: \"c910f899-63b6-466d-adaa-b66ed83517fa\") " pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.565630 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.565609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ckvf\" (UniqueName: \"kubernetes.io/projected/f34e57e5-2188-471a-91ff-92b558e41ea6-kube-api-access-9ckvf\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:21.645803 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.645768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hr5d"
Jul 09 17:40:21.961145 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.960952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:21.961334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.960952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:21.964491 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.964465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:21.964610 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.964494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:21.964610 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:21.964576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f99f5\""
Jul 09 17:40:22.057020 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.056976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:22.057213 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:22.057098    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:22.057213 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:22.057200    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:23.057175267 +0000 UTC m=+36.660060720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:22.479762 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.479728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hr5d"]
Jul 09 17:40:22.560048 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:40:22.560018    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc910f899_63b6_466d_adaa_b66ed83517fa.slice/crio-9d2cdae0560b5a1ba5718fb0c94dbbba3e5f2e19a1e82ec279aaafe0bd301961 WatchSource:0}: Error finding container 9d2cdae0560b5a1ba5718fb0c94dbbba3e5f2e19a1e82ec279aaafe0bd301961: Status 404 returned error can't find the container with id 9d2cdae0560b5a1ba5718fb0c94dbbba3e5f2e19a1e82ec279aaafe0bd301961
Jul 09 17:40:22.961540 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.961514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:22.965004 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.964981    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:22.966330 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.966311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:22.966448 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:22.966314    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9rs28\""
Jul 09 17:40:23.064692 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:23.064659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:23.064828 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:23.064812    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:23.064891 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:23.064881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:25.064865111 +0000 UTC m=+38.667750548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:23.144843 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:23.144808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hr5d" event={"ID":"c910f899-63b6-466d-adaa-b66ed83517fa","Type":"ContainerStarted","Data":"9d2cdae0560b5a1ba5718fb0c94dbbba3e5f2e19a1e82ec279aaafe0bd301961"}
Jul 09 17:40:23.148848 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:23.148746    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="f74eb609d62bfd74ef1e12d2f696233836c345bfc1d2f06a68feb3c1d27fc010" exitCode=0
Jul 09 17:40:23.148848 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:23.148772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"f74eb609d62bfd74ef1e12d2f696233836c345bfc1d2f06a68feb3c1d27fc010"}
Jul 09 17:40:24.154593 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:24.154547    2577 generic.go:356] "Generic (PLEG): container finished" podID="69b094cf-1239-4c18-9511-db2b7c29c5b8" containerID="0181e78c0920c099717d564251f0106b0cff643997b467f7a322de48259bbcec" exitCode=0
Jul 09 17:40:24.155013 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:24.154619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerDied","Data":"0181e78c0920c099717d564251f0106b0cff643997b467f7a322de48259bbcec"}
Jul 09 17:40:25.080156 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:25.080122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:25.080293 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:25.080275    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:25.080349 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:25.080339    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:29.080324542 +0000 UTC m=+42.683209975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:25.157993 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:25.157949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hr5d" event={"ID":"c910f899-63b6-466d-adaa-b66ed83517fa","Type":"ContainerStarted","Data":"bc942a058d5732b606bbd2f2c31f932613970802dc632f7849b3a93ce09eb89d"}
Jul 09 17:40:25.161614 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:25.161584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-27sfj" event={"ID":"69b094cf-1239-4c18-9511-db2b7c29c5b8","Type":"ContainerStarted","Data":"f8a09d8e8c1b21ca111dd6f36d4b4eacbfe4d2e4af130a56a007fb824547436e"}
Jul 09 17:40:25.224185 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:25.224134    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5hr5d" podStartSLOduration=2.012964564 podStartE2EDuration="4.224118108s" totalImagesPullingTime="2.211153544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:25.189206747 +0000 UTC m=+38.792092202" watchObservedRunningTime="2026-07-09 17:40:25.224118108 +0000 UTC m=+38.827003542"
Jul 09 17:40:29.107974 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:29.107924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:29.108519 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:29.108077    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:29.108519 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:29.108145    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:37.108127391 +0000 UTC m=+50.711012830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:37.164069 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:37.164030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:37.164565 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:37.164146    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:37.164565 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:37.164201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:53.16418749 +0000 UTC m=+66.767072924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:47.136510 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:47.136484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txgxp"
Jul 09 17:40:47.170682 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:47.170638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-27sfj" podStartSLOduration=30.439774313 podStartE2EDuration="1m0.17062224s" totalImagesPullingTime="29.730847927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:25.22607781 +0000 UTC m=+38.828963268" watchObservedRunningTime="2026-07-09 17:40:47.17062224 +0000 UTC m=+60.773507696"
Jul 09 17:40:52.672577 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.672523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:40:52.672577 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.672583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:52.673061 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.672619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:52.676194 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.676174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:52.676250 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.676236    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:52.676296 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.676246    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:52.683700 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:52.683680    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 17:40:52.683760 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:52.683748    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs podName:30a61f7d-2778-48d8-b750-6570eccbf1b5 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:56.683728659 +0000 UTC m=+130.286614114 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs") pod "network-metrics-daemon-dbfs7" (UID: "30a61f7d-2778-48d8-b750-6570eccbf1b5") : secret "metrics-daemon-secret" not found
Jul 09 17:40:52.686586 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.686571    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:52.687405 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.687389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/76cd4c1f-fee0-4f8c-b9ac-026346a096bb-original-pull-secret\") pod \"global-pull-secret-syncer-ss8kv\" (UID: \"76cd4c1f-fee0-4f8c-b9ac-026346a096bb\") " pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:52.695984 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.695957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq58v\" (UniqueName: \"kubernetes.io/projected/018d06aa-ce76-4485-95e0-e46b98c1f69c-kube-api-access-wq58v\") pod \"network-check-target-llc92\" (UID: \"018d06aa-ce76-4485-95e0-e46b98c1f69c\") " pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:52.878110 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.878062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ss8kv"
Jul 09 17:40:52.973233 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.973208    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9rs28\""
Jul 09 17:40:52.980164 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:52.980144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:53.018297 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:53.018241    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ss8kv"]
Jul 09 17:40:53.021864 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:40:53.021839    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cd4c1f_fee0_4f8c_b9ac_026346a096bb.slice/crio-569e7ab38fd3af57d4a767f8defb7c6ac98bd47af3242449e39e201ee9ed53a4 WatchSource:0}: Error finding container 569e7ab38fd3af57d4a767f8defb7c6ac98bd47af3242449e39e201ee9ed53a4: Status 404 returned error can't find the container with id 569e7ab38fd3af57d4a767f8defb7c6ac98bd47af3242449e39e201ee9ed53a4
Jul 09 17:40:53.114329 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:53.114300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llc92"]
Jul 09 17:40:53.117736 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:40:53.117703    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018d06aa_ce76_4485_95e0_e46b98c1f69c.slice/crio-41d7c1de0c63403a7203961910b9cba31935b6b03385028820181796349a8a07 WatchSource:0}: Error finding container 41d7c1de0c63403a7203961910b9cba31935b6b03385028820181796349a8a07: Status 404 returned error can't find the container with id 41d7c1de0c63403a7203961910b9cba31935b6b03385028820181796349a8a07
Jul 09 17:40:53.176491 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:53.176409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:40:53.176622 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:53.176530    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:53.176622 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:40:53.176582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls podName:f34e57e5-2188-471a-91ff-92b558e41ea6 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:25.176567257 +0000 UTC m=+98.779452692 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls") pod "dns-default-6vc6k" (UID: "f34e57e5-2188-471a-91ff-92b558e41ea6") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:53.228939 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:53.228907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llc92" event={"ID":"018d06aa-ce76-4485-95e0-e46b98c1f69c","Type":"ContainerStarted","Data":"41d7c1de0c63403a7203961910b9cba31935b6b03385028820181796349a8a07"}
Jul 09 17:40:53.229930 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:53.229906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ss8kv" event={"ID":"76cd4c1f-fee0-4f8c-b9ac-026346a096bb","Type":"ContainerStarted","Data":"569e7ab38fd3af57d4a767f8defb7c6ac98bd47af3242449e39e201ee9ed53a4"}
Jul 09 17:40:58.245738 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:58.245694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llc92" event={"ID":"018d06aa-ce76-4485-95e0-e46b98c1f69c","Type":"ContainerStarted","Data":"28a55c90cf9e263ad42ded5af89de658fd66651a00ee8e13a48762f850c755ba"}
Jul 09 17:40:58.246203 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:58.245819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:40:58.247143 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:58.247114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ss8kv" event={"ID":"76cd4c1f-fee0-4f8c-b9ac-026346a096bb","Type":"ContainerStarted","Data":"f2c147732c6f156484858e82e84575cd74d3e55267de96bd0ac9dc9f52a33d31"}
Jul 09 17:40:58.263304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:58.263264    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-llc92" podStartSLOduration=66.56847475000001 podStartE2EDuration="1m11.263250084s" totalImagesPullingTime="4.694775334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:58.26287885 +0000 UTC m=+71.865764327" watchObservedRunningTime="2026-07-09 17:40:58.263250084 +0000 UTC m=+71.866135543"
Jul 09 17:40:58.279209 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:40:58.279167    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ss8kv" podStartSLOduration=66.47727961 podStartE2EDuration="1m11.279153456s" totalImagesPullingTime="4.801873846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:58.27881974 +0000 UTC m=+71.881705199" watchObservedRunningTime="2026-07-09 17:40:58.279153456 +0000 UTC m=+71.882038911"
Jul 09 17:41:03.843563 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.843528    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2xvxr"]
Jul 09 17:41:03.845602 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.845580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.848991 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.848964    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 09 17:41:03.848991 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.848974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 17:41:03.848991 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.848961    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 17:41:03.850429 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.850409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 09 17:41:03.850523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.850412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wcfpq\""
Jul 09 17:41:03.856452 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.856429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 09 17:41:03.856949 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.856929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2xvxr"]
Jul 09 17:41:03.949218 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.949372 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef9402b-6459-4239-ad0d-0e5d825bd721-serving-cert\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.949372 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmptk\" (UniqueName: \"kubernetes.io/projected/5ef9402b-6459-4239-ad0d-0e5d825bd721-kube-api-access-tmptk\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.949372 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-snapshots\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.949502 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.949502 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.949408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-tmp\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:03.966020 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.965985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7945b9f8cd-6dlzr"]
Jul 09 17:41:03.967989 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.967974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:03.972230 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.972211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 17:41:03.972728 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.972707    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 17:41:03.972848 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.972830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wxj9q\""
Jul 09 17:41:03.973094 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.973078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 17:41:03.973177 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.973078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 17:41:03.973177 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.973166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 17:41:03.975244 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:03.975229    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 17:41:04.001498 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.001472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7945b9f8cd-6dlzr"]
Jul 09 17:41:04.049854 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-snapshots\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.050022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-stats-auth\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.050022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-tmp\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.049969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmpq\" (UniqueName: \"kubernetes.io/projected/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-kube-api-access-hzmpq\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.050304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.050304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef9402b-6459-4239-ad0d-0e5d825bd721-serving-cert\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmptk\" (UniqueName: \"kubernetes.io/projected/5ef9402b-6459-4239-ad0d-0e5d825bd721-kube-api-access-tmptk\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050304 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-default-certificate\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.050575 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-tmp\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050632 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5ef9402b-6459-4239-ad0d-0e5d825bd721-snapshots\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.050686 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.050655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.051396 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.051377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ef9402b-6459-4239-ad0d-0e5d825bd721-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.052604 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.052587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef9402b-6459-4239-ad0d-0e5d825bd721-serving-cert\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.077645 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.077617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmptk\" (UniqueName: \"kubernetes.io/projected/5ef9402b-6459-4239-ad0d-0e5d825bd721-kube-api-access-tmptk\") pod \"insights-operator-68d9dd94cf-2xvxr\" (UID: \"5ef9402b-6459-4239-ad0d-0e5d825bd721\") " pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.150811 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.150730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.150811 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.150767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-stats-auth\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.150811 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.150787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmpq\" (UniqueName: \"kubernetes.io/projected/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-kube-api-access-hzmpq\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.151026 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.150904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.151026 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.150918    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:04.65089899 +0000 UTC m=+78.253784436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : configmap references non-existent config key: service-ca.crt
Jul 09 17:41:04.151026 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.150987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-default-certificate\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.151148 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.151028    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:41:04.151148 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.151094    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:04.65107595 +0000 UTC m=+78.253961392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : secret "router-metrics-certs-default" not found
Jul 09 17:41:04.153321 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.153295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-stats-auth\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.153534 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.153514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-default-certificate\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.153629 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.153616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr"
Jul 09 17:41:04.153862 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.153842    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"]
Jul 09 17:41:04.156260 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.156241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.161473 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.161453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 17:41:04.161581 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.161480    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 17:41:04.161634 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.161616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2s4mh\""
Jul 09 17:41:04.165405 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.165381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:41:04.165528 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.165482    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 17:41:04.183122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.183089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"]
Jul 09 17:41:04.183652 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.183630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmpq\" (UniqueName: \"kubernetes.io/projected/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-kube-api-access-hzmpq\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.251907 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.251878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.252052 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.251917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.252052 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.252027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvxn\" (UniqueName: \"kubernetes.io/projected/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-kube-api-access-4xvxn\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.279416 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.279387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-2xvxr"]
Jul 09 17:41:04.282545 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:04.282517    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef9402b_6459_4239_ad0d_0e5d825bd721.slice/crio-3f5e2da2f5376f8fa5dcf8ed4a79b9734889f935645b82d2923554da262b84d1 WatchSource:0}: Error finding container 3f5e2da2f5376f8fa5dcf8ed4a79b9734889f935645b82d2923554da262b84d1: Status 404 returned error can't find the container with id 3f5e2da2f5376f8fa5dcf8ed4a79b9734889f935645b82d2923554da262b84d1
Jul 09 17:41:04.352440 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.352408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvxn\" (UniqueName: \"kubernetes.io/projected/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-kube-api-access-4xvxn\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.352617 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.352460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.352617 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.352484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.353266 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.353241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.354763 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.354743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.361032 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.361007    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvxn\" (UniqueName: \"kubernetes.io/projected/7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949-kube-api-access-4xvxn\") pod \"kube-storage-version-migrator-operator-799cb7f958-k2wql\" (UID: \"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.473331 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.473299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"
Jul 09 17:41:04.590009 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.589973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql"]
Jul 09 17:41:04.593207 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:04.593176    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da0fe58_3ce8_475a_9b6a_5cdf6f1d9949.slice/crio-edb0def1db5271939cd56fbc45683d543f017d78a471a44c1af88b6002b640fe WatchSource:0}: Error finding container edb0def1db5271939cd56fbc45683d543f017d78a471a44c1af88b6002b640fe: Status 404 returned error can't find the container with id edb0def1db5271939cd56fbc45683d543f017d78a471a44c1af88b6002b640fe
Jul 09 17:41:04.654467 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.654429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.654634 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:04.654486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:04.654634 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.654586    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:41:04.654634 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.654610    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:05.654591274 +0000 UTC m=+79.257476708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : configmap references non-existent config key: service-ca.crt
Jul 09 17:41:04.654634 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:04.654633    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:05.654625867 +0000 UTC m=+79.257511301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : secret "router-metrics-certs-default" not found
Jul 09 17:41:05.265744 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:05.265688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql" event={"ID":"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949","Type":"ContainerStarted","Data":"edb0def1db5271939cd56fbc45683d543f017d78a471a44c1af88b6002b640fe"}
Jul 09 17:41:05.267264 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:05.267234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr" event={"ID":"5ef9402b-6459-4239-ad0d-0e5d825bd721","Type":"ContainerStarted","Data":"3f5e2da2f5376f8fa5dcf8ed4a79b9734889f935645b82d2923554da262b84d1"}
Jul 09 17:41:05.662438 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:05.662331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:05.662438 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:05.662435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:05.662625 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:05.662511    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:07.662491198 +0000 UTC m=+81.265376648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : configmap references non-existent config key: service-ca.crt
Jul 09 17:41:05.662625 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:05.662539    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:41:05.662625 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:05.662574    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:07.662562456 +0000 UTC m=+81.265447890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : secret "router-metrics-certs-default" not found
Jul 09 17:41:07.274074 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.274036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr" event={"ID":"5ef9402b-6459-4239-ad0d-0e5d825bd721","Type":"ContainerStarted","Data":"9b7006b76443b866add3e74d4cf1fa5470365ab52b124878a7647dd6ea62f984"}
Jul 09 17:41:07.275647 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.275620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql" event={"ID":"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949","Type":"ContainerStarted","Data":"dc626e720f02a74797ef990572269bc92df8b33144525e376f16906899a0ed5e"}
Jul 09 17:41:07.293553 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.293508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr" podStartSLOduration=1.659949749 podStartE2EDuration="4.293494003s" totalImagesPullingTime="2.633544254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:07.29293157 +0000 UTC m=+80.895817028" watchObservedRunningTime="2026-07-09 17:41:07.293494003 +0000 UTC m=+80.896379455"
Jul 09 17:41:07.311402 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.311329    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql" podStartSLOduration=0.985460561 podStartE2EDuration="3.31130978s" totalImagesPullingTime="2.325849219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:07.311107808 +0000 UTC m=+80.913993257" watchObservedRunningTime="2026-07-09 17:41:07.31130978 +0000 UTC m=+80.914195238"
Jul 09 17:41:07.678035 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.677941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:07.678199 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:07.678034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:07.678199 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:07.678107    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:41:07.678199 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:07.678177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.678160412 +0000 UTC m=+85.281045846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : secret "router-metrics-certs-default" not found
Jul 09 17:41:07.678199 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:07.678191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.678185177 +0000 UTC m=+85.281070611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : configmap references non-existent config key: service-ca.crt
Jul 09 17:41:08.028245 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.028203    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-828zp"]
Jul 09 17:41:08.031402 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.031377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.035570 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.035548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n86q2\""
Jul 09 17:41:08.036851 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.036832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 17:41:08.037992 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.037922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 17:41:08.047948 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.047922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-828zp"]
Jul 09 17:41:08.081294 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.081258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.081472 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.081300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b09a43-b99a-40dd-b630-b4e109562963-data-volume\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.081472 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.081441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b09a43-b99a-40dd-b630-b4e109562963-crio-socket\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.081569 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.081467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.081569 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.081503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47m9s\" (UniqueName: \"kubernetes.io/projected/23b09a43-b99a-40dd-b630-b4e109562963-kube-api-access-47m9s\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.181960 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.181842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.181960 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.181877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b09a43-b99a-40dd-b630-b4e109562963-data-volume\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.181995    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.182028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b09a43-b99a-40dd-b630-b4e109562963-crio-socket\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.182055    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:08.682039191 +0000 UTC m=+82.284924628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.182077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.182096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b09a43-b99a-40dd-b630-b4e109562963-crio-socket\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.182095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47m9s\" (UniqueName: \"kubernetes.io/projected/23b09a43-b99a-40dd-b630-b4e109562963-kube-api-access-47m9s\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.182180 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.182147    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:41:08.182466 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.182201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:08.68218416 +0000 UTC m=+82.285069607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : configmap "kube-rbac-proxy" not found
Jul 09 17:41:08.182466 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.182198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b09a43-b99a-40dd-b630-b4e109562963-data-volume\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.199720 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.199690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47m9s\" (UniqueName: \"kubernetes.io/projected/23b09a43-b99a-40dd-b630-b4e109562963-kube-api-access-47m9s\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.272453 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.272416    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"]
Jul 09 17:41:08.274515 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.274483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"
Jul 09 17:41:08.277879 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.277855    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v2fwj\""
Jul 09 17:41:08.278132 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.277857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 17:41:08.278132 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.277884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 17:41:08.283924 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.283902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"]
Jul 09 17:41:08.384498 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.384465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfw8x\" (UniqueName: \"kubernetes.io/projected/9bdd5402-1bab-497d-a86b-d5d5f1543a51-kube-api-access-jfw8x\") pod \"migrator-7f59b54f6c-sgt5h\" (UID: \"9bdd5402-1bab-497d-a86b-d5d5f1543a51\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"
Jul 09 17:41:08.485022 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.484988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfw8x\" (UniqueName: \"kubernetes.io/projected/9bdd5402-1bab-497d-a86b-d5d5f1543a51-kube-api-access-jfw8x\") pod \"migrator-7f59b54f6c-sgt5h\" (UID: \"9bdd5402-1bab-497d-a86b-d5d5f1543a51\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"
Jul 09 17:41:08.493460 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.493431    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfw8x\" (UniqueName: \"kubernetes.io/projected/9bdd5402-1bab-497d-a86b-d5d5f1543a51-kube-api-access-jfw8x\") pod \"migrator-7f59b54f6c-sgt5h\" (UID: \"9bdd5402-1bab-497d-a86b-d5d5f1543a51\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"
Jul 09 17:41:08.584561 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.584475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"
Jul 09 17:41:08.687042 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.687011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.687200 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.687072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:08.687200 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.687168    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:41:08.687313 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.687219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:09.68720397 +0000 UTC m=+83.290089403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : configmap "kube-rbac-proxy" not found
Jul 09 17:41:08.687313 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.687168    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:08.687313 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:08.687263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:09.687256047 +0000 UTC m=+83.290141481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:08.706329 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:08.706301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h"]
Jul 09 17:41:08.709958 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:08.709933    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdd5402_1bab_497d_a86b_d5d5f1543a51.slice/crio-069b31a8afb230221182a6ca46083e4823123a3dae021b211027eb1f89acb146 WatchSource:0}: Error finding container 069b31a8afb230221182a6ca46083e4823123a3dae021b211027eb1f89acb146: Status 404 returned error can't find the container with id 069b31a8afb230221182a6ca46083e4823123a3dae021b211027eb1f89acb146
Jul 09 17:41:09.283409 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:09.283377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h" event={"ID":"9bdd5402-1bab-497d-a86b-d5d5f1543a51","Type":"ContainerStarted","Data":"069b31a8afb230221182a6ca46083e4823123a3dae021b211027eb1f89acb146"}
Jul 09 17:41:09.697693 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:09.697647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:09.697897 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:09.697745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:09.697897 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:09.697830    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:41:09.697897 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:09.697846    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:09.697897 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:09.697889    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.697870482 +0000 UTC m=+85.300755917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : configmap "kube-rbac-proxy" not found
Jul 09 17:41:09.698066 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:09.697923    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.697905375 +0000 UTC m=+85.300790826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:09.795476 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:09.795438    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg4dz_2d7d8c3a-52bd-4089-8af7-90a74d3abfab/dns-node-resolver/0.log"
Jul 09 17:41:10.288338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:10.288256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h" event={"ID":"9bdd5402-1bab-497d-a86b-d5d5f1543a51","Type":"ContainerStarted","Data":"5eabd3eebe4a70fd86b2107a48722e02efe2b18fcda289db154301072f57f946"}
Jul 09 17:41:10.288338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:10.288309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h" event={"ID":"9bdd5402-1bab-497d-a86b-d5d5f1543a51","Type":"ContainerStarted","Data":"b105a8a67fb8493cac02ed66d3ff02e411a54859881f7cedd2d00633defba834"}
Jul 09 17:41:10.306318 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:10.306268    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-sgt5h" podStartSLOduration=1.1386149859999999 podStartE2EDuration="2.30625393s" totalImagesPullingTime="1.167638944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:10.306170241 +0000 UTC m=+83.909055728" watchObservedRunningTime="2026-07-09 17:41:10.30625393 +0000 UTC m=+83.909139387"
Jul 09 17:41:10.996243 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:10.996205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-757rc_6a13b98e-d5f8-46db-8552-838b46760281/node-ca/0.log"
Jul 09 17:41:11.595766 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:11.595723    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hr5d_c910f899-63b6-466d-adaa-b66ed83517fa/serve-healthcheck-canary/0.log"
Jul 09 17:41:11.714491 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:11.714453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:11.714655 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:11.714502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:11.714655 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:11.714544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:11.714655 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:11.714585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:11.714655 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714597    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:15.714647373 +0000 UTC m=+89.317532811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714663    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714660    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714694    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:19.714682989 +0000 UTC m=+93.317568422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : configmap references non-existent config key: service-ca.crt
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714713    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs podName:b0fe08a2-2320-46a9-8b03-08f8af1a8c67 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:19.714706769 +0000 UTC m=+93.317592202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs") pod "router-default-7945b9f8cd-6dlzr" (UID: "b0fe08a2-2320-46a9-8b03-08f8af1a8c67") : secret "router-metrics-certs-default" not found
Jul 09 17:41:11.714785 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:11.714723    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:15.714717785 +0000 UTC m=+89.317603219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : configmap "kube-rbac-proxy" not found
Jul 09 17:41:15.750111 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:15.750058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:15.750557 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:15.750139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:15.750557 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:15.750209    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:41:15.750557 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:15.750211    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:15.750557 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:15.750261    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:23.75024838 +0000 UTC m=+97.353133814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : configmap "kube-rbac-proxy" not found
Jul 09 17:41:15.750557 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:41:15.750274    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls podName:23b09a43-b99a-40dd-b630-b4e109562963 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:23.75026828 +0000 UTC m=+97.353153714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls") pod "insights-runtime-extractor-828zp" (UID: "23b09a43-b99a-40dd-b630-b4e109562963") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:41:19.781633 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:19.781582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:19.782152 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:19.781654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:19.782455 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:19.782432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-service-ca-bundle\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:19.783996 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:19.783973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fe08a2-2320-46a9-8b03-08f8af1a8c67-metrics-certs\") pod \"router-default-7945b9f8cd-6dlzr\" (UID: \"b0fe08a2-2320-46a9-8b03-08f8af1a8c67\") " pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:19.876003 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:19.875966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:20.005444 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.005416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7945b9f8cd-6dlzr"]
Jul 09 17:41:20.009063 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:20.009036    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fe08a2_2320_46a9_8b03_08f8af1a8c67.slice/crio-b987a4a9caf6f4b49542954b04becf5e67f09ff22d14e117b2ec6bd9a46ac5b8 WatchSource:0}: Error finding container b987a4a9caf6f4b49542954b04becf5e67f09ff22d14e117b2ec6bd9a46ac5b8: Status 404 returned error can't find the container with id b987a4a9caf6f4b49542954b04becf5e67f09ff22d14e117b2ec6bd9a46ac5b8
Jul 09 17:41:20.318501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.318417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr" event={"ID":"b0fe08a2-2320-46a9-8b03-08f8af1a8c67","Type":"ContainerStarted","Data":"898b6132a8aaf7d96cebc5f0411dd3264ad943138b811941805a6ef7a69c1276"}
Jul 09 17:41:20.318501 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.318464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr" event={"ID":"b0fe08a2-2320-46a9-8b03-08f8af1a8c67","Type":"ContainerStarted","Data":"b987a4a9caf6f4b49542954b04becf5e67f09ff22d14e117b2ec6bd9a46ac5b8"}
Jul 09 17:41:20.340217 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.340170    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr" podStartSLOduration=17.340153876 podStartE2EDuration="17.340153876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:20.339058651 +0000 UTC m=+93.941944109" watchObservedRunningTime="2026-07-09 17:41:20.340153876 +0000 UTC m=+93.943039332"
Jul 09 17:41:20.876919 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.876881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:20.879512 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:20.879487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:21.321724 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:21.321691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:21.322976 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:21.322958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7945b9f8cd-6dlzr"
Jul 09 17:41:23.810849 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:23.810814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:23.811230 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:23.810874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:23.811497 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:23.811478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b09a43-b99a-40dd-b630-b4e109562963-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:23.813193 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:23.813172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b09a43-b99a-40dd-b630-b4e109562963-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-828zp\" (UID: \"23b09a43-b99a-40dd-b630-b4e109562963\") " pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:23.940679 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:23.940645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-828zp"
Jul 09 17:41:24.063537 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:24.063458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-828zp"]
Jul 09 17:41:24.066676 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:24.066640    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b09a43_b99a_40dd_b630_b4e109562963.slice/crio-5e13044ccd04dd5c47ce82b8b1e39ca69dab8c2fcf140235718e0d57c0d72c03 WatchSource:0}: Error finding container 5e13044ccd04dd5c47ce82b8b1e39ca69dab8c2fcf140235718e0d57c0d72c03: Status 404 returned error can't find the container with id 5e13044ccd04dd5c47ce82b8b1e39ca69dab8c2fcf140235718e0d57c0d72c03
Jul 09 17:41:24.331477 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:24.331388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-828zp" event={"ID":"23b09a43-b99a-40dd-b630-b4e109562963","Type":"ContainerStarted","Data":"5e13044ccd04dd5c47ce82b8b1e39ca69dab8c2fcf140235718e0d57c0d72c03"}
Jul 09 17:41:25.222702 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:25.222663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:41:25.225192 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:25.225158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f34e57e5-2188-471a-91ff-92b558e41ea6-metrics-tls\") pod \"dns-default-6vc6k\" (UID: \"f34e57e5-2188-471a-91ff-92b558e41ea6\") " pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:41:25.523343 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:25.523306    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lljhl\""
Jul 09 17:41:25.530910 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:25.530877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:41:25.668129 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:25.668097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vc6k"]
Jul 09 17:41:27.425730 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:27.425683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34e57e5_2188_471a_91ff_92b558e41ea6.slice/crio-3b7eb27cdb053207db8fcd986499ebc9ed0b01f722f3ca29683bfdfab2ec9767 WatchSource:0}: Error finding container 3b7eb27cdb053207db8fcd986499ebc9ed0b01f722f3ca29683bfdfab2ec9767: Status 404 returned error can't find the container with id 3b7eb27cdb053207db8fcd986499ebc9ed0b01f722f3ca29683bfdfab2ec9767
Jul 09 17:41:28.346558 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:28.346514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-828zp" event={"ID":"23b09a43-b99a-40dd-b630-b4e109562963","Type":"ContainerStarted","Data":"e0255cfddc73c90b7b1ca601d20efb6ac5a97ba749a14282285e4fafe1818e5e"}
Jul 09 17:41:28.347723 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:28.347680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vc6k" event={"ID":"f34e57e5-2188-471a-91ff-92b558e41ea6","Type":"ContainerStarted","Data":"3b7eb27cdb053207db8fcd986499ebc9ed0b01f722f3ca29683bfdfab2ec9767"}
Jul 09 17:41:29.252680 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.252090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-llc92"
Jul 09 17:41:29.321570 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.321545    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wklcb"]
Jul 09 17:41:29.329534 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.329504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.333172 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.333109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nwtcl\""
Jul 09 17:41:29.333172 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.333131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 17:41:29.334310 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.334289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 17:41:29.349477 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.349412    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wklcb"]
Jul 09 17:41:29.353771 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.353603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vc6k" event={"ID":"f34e57e5-2188-471a-91ff-92b558e41ea6","Type":"ContainerStarted","Data":"508be5c8c008a7746bcc4820154706dc188e47f31cdef2c768e5a675cef0fc69"}
Jul 09 17:41:29.353771 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.353669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vc6k" event={"ID":"f34e57e5-2188-471a-91ff-92b558e41ea6","Type":"ContainerStarted","Data":"6c40e8f9fd831509788cf40f469a26e426f0ddb13788409c5d2d4dcad292c2bb"}
Jul 09 17:41:29.353771 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.353718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:41:29.355691 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.355653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-828zp" event={"ID":"23b09a43-b99a-40dd-b630-b4e109562963","Type":"ContainerStarted","Data":"19ed5b4c55b95eb59e91dc6e1ddfebe10fc8e574f8090ca8883e32ac9df1d716"}
Jul 09 17:41:29.433164 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.433131    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"]
Jul 09 17:41:29.440165 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.440140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.446111 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.445907    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 17:41:29.446111 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.445985    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 17:41:29.446324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.446307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 17:41:29.446769 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.446751    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 17:41:29.452438 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.452410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"]
Jul 09 17:41:29.453864 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.453824    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6vc6k" podStartSLOduration=66.780517579 podStartE2EDuration="1m8.453810663s" totalImagesPullingTime="1.673293084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:29.452874196 +0000 UTC m=+103.055759680" watchObservedRunningTime="2026-07-09 17:41:29.453810663 +0000 UTC m=+103.056696121"
Jul 09 17:41:29.456799 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.456778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e04c5bee-815a-458c-a969-5a7cdbab06be-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.456904 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.456854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e04c5bee-815a-458c-a969-5a7cdbab06be-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.481728 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.481652    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-ls8mz"]
Jul 09 17:41:29.488218 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.488189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:29.491557 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.491536    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b77wn\""
Jul 09 17:41:29.492093 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.492072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 17:41:29.492213 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.492194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 17:41:29.516090 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.516063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-ls8mz"]
Jul 09 17:41:29.534747 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.534718    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-784764f4c7-kjnbw"]
Jul 09 17:41:29.541661 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.541631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.546744 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.546722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 17:41:29.546870 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.546726    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 17:41:29.547616 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.547430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c6j4v\""
Jul 09 17:41:29.551265 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.551094    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 17:41:29.553452 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.553430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 17:41:29.557143 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebaaa45-c562-4514-883a-039cbd22441d-tmp\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.557232 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e04c5bee-815a-458c-a969-5a7cdbab06be-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.557232 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbkj\" (UniqueName: \"kubernetes.io/projected/8ebaaa45-c562-4514-883a-039cbd22441d-kube-api-access-zpbkj\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.557350 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ebaaa45-c562-4514-883a-039cbd22441d-klusterlet-config\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.557350 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e04c5bee-815a-458c-a969-5a7cdbab06be-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.557510 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kf5l\" (UniqueName: \"kubernetes.io/projected/37a82a7b-936f-4f81-943d-a1fcc555143a-kube-api-access-4kf5l\") pod \"downloads-68c9cbb497-ls8mz\" (UID: \"37a82a7b-936f-4f81-943d-a1fcc555143a\") " pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:29.557933 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.557910    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e04c5bee-815a-458c-a969-5a7cdbab06be-nginx-conf\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.559975 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.559921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e04c5bee-815a-458c-a969-5a7cdbab06be-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-wklcb\" (UID: \"e04c5bee-815a-458c-a969-5a7cdbab06be\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.563963 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.563941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-784764f4c7-kjnbw"]
Jul 09 17:41:29.641164 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.641072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb"
Jul 09 17:41:29.658001 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.657967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-bound-sa-token\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.658106 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbkj\" (UniqueName: \"kubernetes.io/projected/8ebaaa45-c562-4514-883a-039cbd22441d-kube-api-access-zpbkj\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.658461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-image-registry-private-configuration\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.658461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnfq\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-kube-api-access-9gnfq\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.658461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ebaaa45-c562-4514-883a-039cbd22441d-klusterlet-config\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.658461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kf5l\" (UniqueName: \"kubernetes.io/projected/37a82a7b-936f-4f81-943d-a1fcc555143a-kube-api-access-4kf5l\") pod \"downloads-68c9cbb497-ls8mz\" (UID: \"37a82a7b-936f-4f81-943d-a1fcc555143a\") " pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:29.658461 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-registry-tls\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.659071 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a1675f-4846-4a5b-a92e-b61241620048-ca-trust-extracted\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.659071 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-registry-certificates\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.659071 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-installation-pull-secrets\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.659071 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-trusted-ca\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.659071 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.658883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebaaa45-c562-4514-883a-039cbd22441d-tmp\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.659286 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.659246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ebaaa45-c562-4514-883a-039cbd22441d-tmp\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.661393 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.661346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8ebaaa45-c562-4514-883a-039cbd22441d-klusterlet-config\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.674431 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.674406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbkj\" (UniqueName: \"kubernetes.io/projected/8ebaaa45-c562-4514-883a-039cbd22441d-kube-api-access-zpbkj\") pod \"klusterlet-addon-workmgr-f46cf5644-gwdhq\" (UID: \"8ebaaa45-c562-4514-883a-039cbd22441d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.675998 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.675973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kf5l\" (UniqueName: \"kubernetes.io/projected/37a82a7b-936f-4f81-943d-a1fcc555143a-kube-api-access-4kf5l\") pod \"downloads-68c9cbb497-ls8mz\" (UID: \"37a82a7b-936f-4f81-943d-a1fcc555143a\") " pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:29.751812 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.751778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:29.759851 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.759803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnfq\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-kube-api-access-9gnfq\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.759995 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.759881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-registry-tls\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.759995 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.759909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a1675f-4846-4a5b-a92e-b61241620048-ca-trust-extracted\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.759995 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.759950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-registry-certificates\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.759995 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.759979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-installation-pull-secrets\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.760206 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.760018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-trusted-ca\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.760206 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.760065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-bound-sa-token\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.760206 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.760117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-image-registry-private-configuration\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.760748 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.760595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d4a1675f-4846-4a5b-a92e-b61241620048-ca-trust-extracted\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.761237 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.761211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-registry-certificates\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.761713 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.761673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d4a1675f-4846-4a5b-a92e-b61241620048-trusted-ca\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.763649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.763596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-registry-tls\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.764242 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.764216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-image-registry-private-configuration\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.764673 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.764383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d4a1675f-4846-4a5b-a92e-b61241620048-installation-pull-secrets\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.772052 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.772017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-bound-sa-token\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.779825 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.779807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnfq\" (UniqueName: \"kubernetes.io/projected/d4a1675f-4846-4a5b-a92e-b61241620048-kube-api-access-9gnfq\") pod \"image-registry-784764f4c7-kjnbw\" (UID: \"d4a1675f-4846-4a5b-a92e-b61241620048\") " pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.797469 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.797403    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-wklcb"]
Jul 09 17:41:29.798176 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.798037    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:29.803425 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:29.803401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04c5bee_815a_458c_a969_5a7cdbab06be.slice/crio-0804840f3fcc848fd134e731fa78754d3f6bf7df6d59502a863916232123a3be WatchSource:0}: Error finding container 0804840f3fcc848fd134e731fa78754d3f6bf7df6d59502a863916232123a3be: Status 404 returned error can't find the container with id 0804840f3fcc848fd134e731fa78754d3f6bf7df6d59502a863916232123a3be
Jul 09 17:41:29.854143 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.854095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:29.954162 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:29.954077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"]
Jul 09 17:41:29.958124 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:29.957732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebaaa45_c562_4514_883a_039cbd22441d.slice/crio-040d297b1439b171a553e9490908083388765c8fe4cdbc65586d432d790eb8e6 WatchSource:0}: Error finding container 040d297b1439b171a553e9490908083388765c8fe4cdbc65586d432d790eb8e6: Status 404 returned error can't find the container with id 040d297b1439b171a553e9490908083388765c8fe4cdbc65586d432d790eb8e6
Jul 09 17:41:30.002206 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.002173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-ls8mz"]
Jul 09 17:41:30.006387 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:30.006326    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a82a7b_936f_4f81_943d_a1fcc555143a.slice/crio-fb56708536dd365692dd971c7f58d991a2e4f6bc6f86e44247b0626398c64598 WatchSource:0}: Error finding container fb56708536dd365692dd971c7f58d991a2e4f6bc6f86e44247b0626398c64598: Status 404 returned error can't find the container with id fb56708536dd365692dd971c7f58d991a2e4f6bc6f86e44247b0626398c64598
Jul 09 17:41:30.062486 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.062435    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-784764f4c7-kjnbw"]
Jul 09 17:41:30.066892 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:30.066862    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a1675f_4846_4a5b_a92e_b61241620048.slice/crio-a1e92230c764b284ada28fcd244c88fade4079c20d117f7e7b84552795fbca35 WatchSource:0}: Error finding container a1e92230c764b284ada28fcd244c88fade4079c20d117f7e7b84552795fbca35: Status 404 returned error can't find the container with id a1e92230c764b284ada28fcd244c88fade4079c20d117f7e7b84552795fbca35
Jul 09 17:41:30.360931 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.360891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb" event={"ID":"e04c5bee-815a-458c-a969-5a7cdbab06be","Type":"ContainerStarted","Data":"0804840f3fcc848fd134e731fa78754d3f6bf7df6d59502a863916232123a3be"}
Jul 09 17:41:30.362246 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.362222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw" event={"ID":"d4a1675f-4846-4a5b-a92e-b61241620048","Type":"ContainerStarted","Data":"ec0e8c70c288426d9e5b808f59a5fcc3c2bd0a77a955661335ed78931e7d6942"}
Jul 09 17:41:30.362338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.362262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw" event={"ID":"d4a1675f-4846-4a5b-a92e-b61241620048","Type":"ContainerStarted","Data":"a1e92230c764b284ada28fcd244c88fade4079c20d117f7e7b84552795fbca35"}
Jul 09 17:41:30.362338 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.362301    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:30.363294 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.363271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-ls8mz" event={"ID":"37a82a7b-936f-4f81-943d-a1fcc555143a","Type":"ContainerStarted","Data":"fb56708536dd365692dd971c7f58d991a2e4f6bc6f86e44247b0626398c64598"}
Jul 09 17:41:30.364238 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.364220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq" event={"ID":"8ebaaa45-c562-4514-883a-039cbd22441d","Type":"ContainerStarted","Data":"040d297b1439b171a553e9490908083388765c8fe4cdbc65586d432d790eb8e6"}
Jul 09 17:41:30.384879 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:30.384824    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw" podStartSLOduration=1.384803897 podStartE2EDuration="1.384803897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:30.383647317 +0000 UTC m=+103.986532773" watchObservedRunningTime="2026-07-09 17:41:30.384803897 +0000 UTC m=+103.987689358"
Jul 09 17:41:32.377308 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:32.377213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-828zp" event={"ID":"23b09a43-b99a-40dd-b630-b4e109562963","Type":"ContainerStarted","Data":"fb4fbf33ef2c0e9f42b5cb8af1758ae4ad7ce88f5e42c845e0559e1a24da61d6"}
Jul 09 17:41:32.399445 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:32.398337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-828zp" podStartSLOduration=17.630400281 podStartE2EDuration="25.39831604s" totalImagesPullingTime="7.767915759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:41:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:32.397549933 +0000 UTC m=+106.000435419" watchObservedRunningTime="2026-07-09 17:41:32.39831604 +0000 UTC m=+106.001201497"
Jul 09 17:41:34.387250 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:34.387147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb" event={"ID":"e04c5bee-815a-458c-a969-5a7cdbab06be","Type":"ContainerStarted","Data":"df8792f8ed6ba579b2a7314a1ead03863dc36c9ef797ae5ca1cd7918d0b01a7f"}
Jul 09 17:41:34.404971 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:34.404922    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-wklcb" podStartSLOduration=1.345803059 podStartE2EDuration="5.404906786s" totalImagesPullingTime="4.059103727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:34.403354834 +0000 UTC m=+108.006240290" watchObservedRunningTime="2026-07-09 17:41:34.404906786 +0000 UTC m=+108.007792282"
Jul 09 17:41:36.396162 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:36.396112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq" event={"ID":"8ebaaa45-c562-4514-883a-039cbd22441d","Type":"ContainerStarted","Data":"a7ab513899937a9e120c89196417ef668730ea1ec62c6e29e36a152c849c18e6"}
Jul 09 17:41:36.396608 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:36.396269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:36.397955 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:36.397934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq"
Jul 09 17:41:36.414929 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:36.414846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f46cf5644-gwdhq" podStartSLOduration=1.211481836 podStartE2EDuration="7.414830256s" totalImagesPullingTime="6.20334842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:36.414302778 +0000 UTC m=+110.017188234" watchObservedRunningTime="2026-07-09 17:41:36.414830256 +0000 UTC m=+110.017715713"
Jul 09 17:41:37.690139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.690102    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"]
Jul 09 17:41:37.692459 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.692438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.695859 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.695830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 17:41:37.696059 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.696042    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 17:41:37.696148 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.696090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 17:41:37.697505 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.697488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 17:41:37.697607 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.697541    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hswbx\""
Jul 09 17:41:37.697607 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.697544    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 17:41:37.710056 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.710031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"]
Jul 09 17:41:37.730617 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.730595    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r8zcp"]
Jul 09 17:41:37.733590 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.733569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.736620 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.736600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9krtl\""
Jul 09 17:41:37.736620 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.736626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 17:41:37.736808 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.736714    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 17:41:37.737046 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.737028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 17:41:37.832965 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.832923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-tls\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.832972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-root\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-textfile\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-metrics-client-ca\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.833324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/596413e4-cb93-4eb1-b1c4-9fde83ed4841-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.833324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbrt\" (UniqueName: \"kubernetes.io/projected/596413e4-cb93-4eb1-b1c4-9fde83ed4841-kube-api-access-fjbrt\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.833324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vdk\" (UniqueName: \"kubernetes.io/projected/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-kube-api-access-48vdk\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833324 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-wtmp\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833547 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-sys\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833547 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.833547 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.833422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.934384 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-tls\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-root\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-textfile\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-metrics-client-ca\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/596413e4-cb93-4eb1-b1c4-9fde83ed4841-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.934555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-root\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbrt\" (UniqueName: \"kubernetes.io/projected/596413e4-cb93-4eb1-b1c4-9fde83ed4841-kube-api-access-fjbrt\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vdk\" (UniqueName: \"kubernetes.io/projected/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-kube-api-access-48vdk\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-wtmp\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-sys\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.934905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.934829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-textfile\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.935259 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.935195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.935313 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.935260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/596413e4-cb93-4eb1-b1c4-9fde83ed4841-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.935386 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.935343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-sys\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.935386 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.935356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-wtmp\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.935502 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.935425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-metrics-client-ca\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.937013 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.936973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.937013 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.936994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.937156 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.937031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/596413e4-cb93-4eb1-b1c4-9fde83ed4841-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:37.937191 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.937171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-node-exporter-tls\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.943310 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.943243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vdk\" (UniqueName: \"kubernetes.io/projected/9c1fd3e8-f2b9-405e-9b65-19fde324d14f-kube-api-access-48vdk\") pod \"node-exporter-r8zcp\" (UID: \"9c1fd3e8-f2b9-405e-9b65-19fde324d14f\") " pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:37.943310 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:37.943252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbrt\" (UniqueName: \"kubernetes.io/projected/596413e4-cb93-4eb1-b1c4-9fde83ed4841-kube-api-access-fjbrt\") pod \"openshift-state-metrics-84bd68b896-jhrdz\" (UID: \"596413e4-cb93-4eb1-b1c4-9fde83ed4841\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:38.001956 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.001919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"
Jul 09 17:41:38.045058 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.045006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8zcp"
Jul 09 17:41:38.056927 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:38.056881    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1fd3e8_f2b9_405e_9b65_19fde324d14f.slice/crio-05158931424dac97bf9baf712a1fc55320ba0cdc4c562b1b40b2c3a64b0c52d3 WatchSource:0}: Error finding container 05158931424dac97bf9baf712a1fc55320ba0cdc4c562b1b40b2c3a64b0c52d3: Status 404 returned error can't find the container with id 05158931424dac97bf9baf712a1fc55320ba0cdc4c562b1b40b2c3a64b0c52d3
Jul 09 17:41:38.149748 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.149712    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz"]
Jul 09 17:41:38.153757 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:38.153735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596413e4_cb93_4eb1_b1c4_9fde83ed4841.slice/crio-4df1ae275affe2f7749efba6acab96bb01c5e38b67a234ee667a53a27a4697c7 WatchSource:0}: Error finding container 4df1ae275affe2f7749efba6acab96bb01c5e38b67a234ee667a53a27a4697c7: Status 404 returned error can't find the container with id 4df1ae275affe2f7749efba6acab96bb01c5e38b67a234ee667a53a27a4697c7
Jul 09 17:41:38.405215 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.405159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8zcp" event={"ID":"9c1fd3e8-f2b9-405e-9b65-19fde324d14f","Type":"ContainerStarted","Data":"05158931424dac97bf9baf712a1fc55320ba0cdc4c562b1b40b2c3a64b0c52d3"}
Jul 09 17:41:38.407390 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.407345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz" event={"ID":"596413e4-cb93-4eb1-b1c4-9fde83ed4841","Type":"ContainerStarted","Data":"6b2dbc3b76ae20cb6d818ad452664f6f587f7cb16a90c87421805de8905a3f17"}
Jul 09 17:41:38.407523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.407417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz" event={"ID":"596413e4-cb93-4eb1-b1c4-9fde83ed4841","Type":"ContainerStarted","Data":"fd5d1ff957e48b8cfb0eded2289e679de460d02c398c5dd6806d64337bf7544e"}
Jul 09 17:41:38.407523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:38.407450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz" event={"ID":"596413e4-cb93-4eb1-b1c4-9fde83ed4841","Type":"ContainerStarted","Data":"4df1ae275affe2f7749efba6acab96bb01c5e38b67a234ee667a53a27a4697c7"}
Jul 09 17:41:39.367213 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:39.367177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vc6k"
Jul 09 17:41:40.416956 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.416915    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c1fd3e8-f2b9-405e-9b65-19fde324d14f" containerID="79424fc9d64f026684967ea02b04c886d40b99cb66e9947e5d4ebe94193e46a5" exitCode=0
Jul 09 17:41:40.417486 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.417025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8zcp" event={"ID":"9c1fd3e8-f2b9-405e-9b65-19fde324d14f","Type":"ContainerDied","Data":"79424fc9d64f026684967ea02b04c886d40b99cb66e9947e5d4ebe94193e46a5"}
Jul 09 17:41:40.419190 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.419169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz" event={"ID":"596413e4-cb93-4eb1-b1c4-9fde83ed4841","Type":"ContainerStarted","Data":"e30fd1965fbdf6296fd741735aedb9ebc494ba153cd81129625131d8b0b27365"}
Jul 09 17:41:40.454711 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.454662    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-jhrdz" podStartSLOduration=2.143263622 podStartE2EDuration="3.454646848s" totalImagesPullingTime="1.311383226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:40.453765326 +0000 UTC m=+114.056650784" watchObservedRunningTime="2026-07-09 17:41:40.454646848 +0000 UTC m=+114.057532282"
Jul 09 17:41:40.636646 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.636611    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"]
Jul 09 17:41:40.639599 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.639573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.642712 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.642687    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 09 17:41:40.642855 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.642825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 09 17:41:40.643388 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.643128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 09 17:41:40.643388 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.643153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wrljk\""
Jul 09 17:41:40.643388 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.643170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 09 17:41:40.643388 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.643170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ds8t54icum9md\""
Jul 09 17:41:40.643388 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.643169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 09 17:41:40.653042 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.652965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"]
Jul 09 17:41:40.761634 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45rq\" (UniqueName: \"kubernetes.io/projected/678086a1-b9f4-48d0-a9fd-02a9e943551f-kube-api-access-n45rq\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-grpc-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761946 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678086a1-b9f4-48d0-a9fd-02a9e943551f-metrics-client-ca\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761946 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.761946 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.761881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862628 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45rq\" (UniqueName: \"kubernetes.io/projected/678086a1-b9f4-48d0-a9fd-02a9e943551f-kube-api-access-n45rq\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-grpc-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678086a1-b9f4-48d0-a9fd-02a9e943551f-metrics-client-ca\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.862814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.863059 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.863059 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.862888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.863704 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.863632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678086a1-b9f4-48d0-a9fd-02a9e943551f-metrics-client-ca\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.865823 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.865779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.865941 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.865873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.866119 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.866091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-grpc-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.866394 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.866355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.866589 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.866568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.866897 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.866872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/678086a1-b9f4-48d0-a9fd-02a9e943551f-secret-thanos-querier-tls\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.871719 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.871699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45rq\" (UniqueName: \"kubernetes.io/projected/678086a1-b9f4-48d0-a9fd-02a9e943551f-kube-api-access-n45rq\") pod \"thanos-querier-5cdd574c69-7wwk5\" (UID: \"678086a1-b9f4-48d0-a9fd-02a9e943551f\") " pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:40.951092 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:40.951055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:41.096304 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:41.096270    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678086a1_b9f4_48d0_a9fd_02a9e943551f.slice/crio-29531aedd63240ce0c03d5420656d0b4f66b1dcb14996d1a6e73683b2cbefeb9 WatchSource:0}: Error finding container 29531aedd63240ce0c03d5420656d0b4f66b1dcb14996d1a6e73683b2cbefeb9: Status 404 returned error can't find the container with id 29531aedd63240ce0c03d5420656d0b4f66b1dcb14996d1a6e73683b2cbefeb9
Jul 09 17:41:41.097231 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.097205    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"]
Jul 09 17:41:41.425009 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.424916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8zcp" event={"ID":"9c1fd3e8-f2b9-405e-9b65-19fde324d14f","Type":"ContainerStarted","Data":"2177f84bec26191c60f4a4517f4faa74e64d818b4b1121a327763ce4457ce11b"}
Jul 09 17:41:41.425009 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.424980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8zcp" event={"ID":"9c1fd3e8-f2b9-405e-9b65-19fde324d14f","Type":"ContainerStarted","Data":"004513d924952de781154d96594a6f5f859f00fb5db30df29736200c9a7ccbc3"}
Jul 09 17:41:41.426128 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.426102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"29531aedd63240ce0c03d5420656d0b4f66b1dcb14996d1a6e73683b2cbefeb9"}
Jul 09 17:41:41.444636 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.444586    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r8zcp" podStartSLOduration=2.842344991 podStartE2EDuration="4.44457142s" totalImagesPullingTime="1.52825848s" totalInitContainerRuntime="73.967949ms" isStatefulPod=true podCreationTimestamp="2026-07-09 17:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:41.443353785 +0000 UTC m=+115.046239240" watchObservedRunningTime="2026-07-09 17:41:41.44457142 +0000 UTC m=+115.047456875"
Jul 09 17:41:41.956633 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.956594    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54b878b4fd-7nlvt"]
Jul 09 17:41:41.959122 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.959034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.962113 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.962088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 17:41:41.964204 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.963797    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6gvg1gnmaq2n8\""
Jul 09 17:41:41.964204 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.963832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 17:41:41.964204 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.963977    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 17:41:41.964204 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.964065    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 17:41:41.964204 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.964098    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dxr5g\""
Jul 09 17:41:41.970584 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.970518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54b878b4fd-7nlvt"]
Jul 09 17:41:41.971950 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.971924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972060 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.971986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-tls\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972060 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.972015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv52h\" (UniqueName: \"kubernetes.io/projected/f2372a27-0aaf-457e-81be-d46a7acadaba-kube-api-access-vv52h\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972060 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.972046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-metrics-server-audit-profiles\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972228 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.972097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-client-certs\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972228 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.972160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-client-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:41.972393 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:41.972242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f2372a27-0aaf-457e-81be-d46a7acadaba-audit-log\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.073804 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.073762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-client-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.074000 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.073819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f2372a27-0aaf-457e-81be-d46a7acadaba-audit-log\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.074000 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.073852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-tls\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv52h\" (UniqueName: \"kubernetes.io/projected/f2372a27-0aaf-457e-81be-d46a7acadaba-kube-api-access-vv52h\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-metrics-server-audit-profiles\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074470    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-client-certs\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f2372a27-0aaf-457e-81be-d46a7acadaba-audit-log\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.074656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.075668 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.075448    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f2372a27-0aaf-457e-81be-d46a7acadaba-metrics-server-audit-profiles\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.077559 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.077507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-tls\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.078006 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.077971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-secret-metrics-server-client-certs\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.078863 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.078845    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f2372a27-0aaf-457e-81be-d46a7acadaba-client-ca-bundle\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.083678 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.083658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv52h\" (UniqueName: \"kubernetes.io/projected/f2372a27-0aaf-457e-81be-d46a7acadaba-kube-api-access-vv52h\") pod \"metrics-server-54b878b4fd-7nlvt\" (UID: \"f2372a27-0aaf-457e-81be-d46a7acadaba\") " pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.273140 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.273054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:41:42.437207 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:42.437180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54b878b4fd-7nlvt"]
Jul 09 17:41:42.439902 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:42.439872    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2372a27_0aaf_457e_81be_d46a7acadaba.slice/crio-345ef0ba978cc6610204fba394e5cbbdf5b2005c2668006479caff55c0db55b1 WatchSource:0}: Error finding container 345ef0ba978cc6610204fba394e5cbbdf5b2005c2668006479caff55c0db55b1: Status 404 returned error can't find the container with id 345ef0ba978cc6610204fba394e5cbbdf5b2005c2668006479caff55c0db55b1
Jul 09 17:41:43.437431 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:43.437396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt" event={"ID":"f2372a27-0aaf-457e-81be-d46a7acadaba","Type":"ContainerStarted","Data":"345ef0ba978cc6610204fba394e5cbbdf5b2005c2668006479caff55c0db55b1"}
Jul 09 17:41:44.445553 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:44.445351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"1379d708c8d48c2f26dea8fd16764e12d9c3047a7054555c0921eb0d8c732b00"}
Jul 09 17:41:44.445553 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:44.445453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"148e9b68aed33d86c05fbc71af7006c03a56f1b1c827d77ec61bd2a99581b356"}
Jul 09 17:41:44.445553 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:44.445493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"f07923ee716aaf15cd592d9276db4fd1043ad9c5d3b37651254b2519c0e43b24"}
Jul 09 17:41:44.449520 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:44.448690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt" event={"ID":"f2372a27-0aaf-457e-81be-d46a7acadaba","Type":"ContainerStarted","Data":"2fb773c72141b281f9940a6b4fa27668a67f5661487835034f6af0baf58d2350"}
Jul 09 17:41:44.472219 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:44.472161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt" podStartSLOduration=1.807718564 podStartE2EDuration="3.472141055s" totalImagesPullingTime="1.664422491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:44.470123501 +0000 UTC m=+118.073008956" watchObservedRunningTime="2026-07-09 17:41:44.472141055 +0000 UTC m=+118.075026512"
Jul 09 17:41:51.372595 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:51.372564    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-784764f4c7-kjnbw"
Jul 09 17:41:52.487116 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:52.487089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"f0859c02e339fc35b0730ac5266f991d9b82afe57af93b52ac6f910a2ab9bf4a"}
Jul 09 17:41:53.493380 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.493329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-ls8mz" event={"ID":"37a82a7b-936f-4f81-943d-a1fcc555143a","Type":"ContainerStarted","Data":"48e722ed2609f418024c9b954ec6a085efbf7d21359edbd60fa0f310f949fff2"}
Jul 09 17:41:53.493884 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.493520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:53.497202 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.497171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"1da9d8aecd8754a1c2373284ea87eb98876c1f7709cf54d445b4f8c57fce6331"}
Jul 09 17:41:53.497339 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.497230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" event={"ID":"678086a1-b9f4-48d0-a9fd-02a9e943551f","Type":"ContainerStarted","Data":"16f422abd5912e5c9f0ca5191ad7e9f5329bee2cbba1ff007d822420ce199e3f"}
Jul 09 17:41:53.497339 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.497268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:53.500751 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.500730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-ls8mz"
Jul 09 17:41:53.503356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.503339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5"
Jul 09 17:41:53.517256 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.517203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-ls8mz" podStartSLOduration=2.125860672 podStartE2EDuration="24.517183252s" totalImagesPullingTime="22.39132258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:53.51647495 +0000 UTC m=+127.119360406" watchObservedRunningTime="2026-07-09 17:41:53.517183252 +0000 UTC m=+127.120068709"
Jul 09 17:41:53.561261 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:53.561182    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cdd574c69-7wwk5" podStartSLOduration=2.970846084 podStartE2EDuration="13.561157792s" totalImagesPullingTime="10.590311708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:53.558518999 +0000 UTC m=+127.161404482" watchObservedRunningTime="2026-07-09 17:41:53.561157792 +0000 UTC m=+127.164043248"
Jul 09 17:41:56.714310 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:56.714252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:41:56.717130 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:56.717089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30a61f7d-2778-48d8-b750-6570eccbf1b5-metrics-certs\") pod \"network-metrics-daemon-dbfs7\" (UID: \"30a61f7d-2778-48d8-b750-6570eccbf1b5\") " pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:41:56.775396 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:56.775350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f99f5\""
Jul 09 17:41:56.782980 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:56.782952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbfs7"
Jul 09 17:41:56.968527 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:56.968497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbfs7"]
Jul 09 17:41:56.969953 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:41:56.969913    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a61f7d_2778_48d8_b750_6570eccbf1b5.slice/crio-543c532ee65f364cf3e7485cf571b8805bb5de76028dd6415e3ee3e43cbc2e3b WatchSource:0}: Error finding container 543c532ee65f364cf3e7485cf571b8805bb5de76028dd6415e3ee3e43cbc2e3b: Status 404 returned error can't find the container with id 543c532ee65f364cf3e7485cf571b8805bb5de76028dd6415e3ee3e43cbc2e3b
Jul 09 17:41:57.516075 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:57.516030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbfs7" event={"ID":"30a61f7d-2778-48d8-b750-6570eccbf1b5","Type":"ContainerStarted","Data":"543c532ee65f364cf3e7485cf571b8805bb5de76028dd6415e3ee3e43cbc2e3b"}
Jul 09 17:41:59.528775 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:59.528729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbfs7" event={"ID":"30a61f7d-2778-48d8-b750-6570eccbf1b5","Type":"ContainerStarted","Data":"6ca0e6cea80be7b12f3c3a253fd7f49f2d2d389d40dc8e5e9d628e425d2dcd14"}
Jul 09 17:41:59.529236 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:59.528792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbfs7" event={"ID":"30a61f7d-2778-48d8-b750-6570eccbf1b5","Type":"ContainerStarted","Data":"bfb25abfb52871f716d48ddd15a9f4193fda931ee6deff183fa7319388ca2001"}
Jul 09 17:41:59.561224 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:41:59.561166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dbfs7" podStartSLOduration=131.063223916 podStartE2EDuration="2m12.561145128s" totalImagesPullingTime="1.497921212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:59.551274139 +0000 UTC m=+133.154159600" watchObservedRunningTime="2026-07-09 17:41:59.561145128 +0000 UTC m=+133.164030584"
Jul 09 17:42:02.273701 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:02.273667    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:42:02.274175 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:02.273728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:42:18.601810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.601776    2577 generic.go:356] "Generic (PLEG): container finished" podID="7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949" containerID="dc626e720f02a74797ef990572269bc92df8b33144525e376f16906899a0ed5e" exitCode=0
Jul 09 17:42:18.602282 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.601850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql" event={"ID":"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949","Type":"ContainerDied","Data":"dc626e720f02a74797ef990572269bc92df8b33144525e376f16906899a0ed5e"}
Jul 09 17:42:18.602282 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.602245    2577 scope.go:117] "RemoveContainer" containerID="dc626e720f02a74797ef990572269bc92df8b33144525e376f16906899a0ed5e"
Jul 09 17:42:18.603412 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.603391    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ef9402b-6459-4239-ad0d-0e5d825bd721" containerID="9b7006b76443b866add3e74d4cf1fa5470365ab52b124878a7647dd6ea62f984" exitCode=0
Jul 09 17:42:18.603515 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.603463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr" event={"ID":"5ef9402b-6459-4239-ad0d-0e5d825bd721","Type":"ContainerDied","Data":"9b7006b76443b866add3e74d4cf1fa5470365ab52b124878a7647dd6ea62f984"}
Jul 09 17:42:18.603723 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:18.603709    2577 scope.go:117] "RemoveContainer" containerID="9b7006b76443b866add3e74d4cf1fa5470365ab52b124878a7647dd6ea62f984"
Jul 09 17:42:19.608983 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:19.608947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-2xvxr" event={"ID":"5ef9402b-6459-4239-ad0d-0e5d825bd721","Type":"ContainerStarted","Data":"0cd0a026c65af6968b7530479a2a984541e84f8d4a633fa66f606c96966f4470"}
Jul 09 17:42:19.610886 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:19.610860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-k2wql" event={"ID":"7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949","Type":"ContainerStarted","Data":"6a03c94666e1c4f633f03875a85bd9c693d79eac7fdfa02ba267771e64f1cac9"}
Jul 09 17:42:22.278567 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:22.278532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:42:22.282352 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:42:22.282332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54b878b4fd-7nlvt"
Jul 09 17:44:09.961647 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.961567    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"]
Jul 09 17:44:09.964003 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.963983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:09.968453 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.968428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mh9qk\""
Jul 09 17:44:09.969530 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.969509    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 17:44:09.969814 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.969798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 17:44:09.979307 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:09.979282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"]
Jul 09 17:44:10.076234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.076201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.076434 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.076256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.076434 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.076279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5lh\" (UniqueName: \"kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.177520 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.177472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.177656 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.177541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.177656 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.177571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5lh\" (UniqueName: \"kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.177872 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.177850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.177941 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.177921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.186822 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.186796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5lh\" (UniqueName: \"kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.274096 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.274000    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:10.405523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:10.405490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"]
Jul 09 17:44:10.408567 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:44:10.408540    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d16d427_7996_478e_b49f_daabd93d61aa.slice/crio-3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408 WatchSource:0}: Error finding container 3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408: Status 404 returned error can't find the container with id 3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408
Jul 09 17:44:11.016200 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:11.016156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerStarted","Data":"3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408"}
Jul 09 17:44:17.041988 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:17.041954    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d16d427-7996-478e-b49f-daabd93d61aa" containerID="483905959d0f907b8bdb86b85f7ea3df771667cec83d88cbe465474a8f5f8677" exitCode=0
Jul 09 17:44:17.042402 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:17.042013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerDied","Data":"483905959d0f907b8bdb86b85f7ea3df771667cec83d88cbe465474a8f5f8677"}
Jul 09 17:44:23.065593 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:23.065558    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d16d427-7996-478e-b49f-daabd93d61aa" containerID="1333a89434652050a54cbba32f5f6f95fcdf1b14ed99d3f61d25e4e57010d9f1" exitCode=0
Jul 09 17:44:23.065989 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:23.065616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerDied","Data":"1333a89434652050a54cbba32f5f6f95fcdf1b14ed99d3f61d25e4e57010d9f1"}
Jul 09 17:44:29.092778 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:29.092737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerStarted","Data":"d453a164c2f7b1e05cff5eb58a2178ca14ba183c0d477aa44868260a95b5d4b1"}
Jul 09 17:44:29.113193 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:29.113131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" podStartSLOduration=3.592802966 podStartE2EDuration="20.113108828s" totalImagesPullingTime="16.520305862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:44:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:29.111856512 +0000 UTC m=+282.714741970" watchObservedRunningTime="2026-07-09 17:44:29.113108828 +0000 UTC m=+282.715994285"
Jul 09 17:44:30.099092 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:30.099055    2577 generic.go:356] "Generic (PLEG): container finished" podID="8d16d427-7996-478e-b49f-daabd93d61aa" containerID="d453a164c2f7b1e05cff5eb58a2178ca14ba183c0d477aa44868260a95b5d4b1" exitCode=0
Jul 09 17:44:30.099507 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:30.099140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerDied","Data":"d453a164c2f7b1e05cff5eb58a2178ca14ba183c0d477aa44868260a95b5d4b1"}
Jul 09 17:44:31.247530 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.247505    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:31.366622 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.366533    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle\") pod \"8d16d427-7996-478e-b49f-daabd93d61aa\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") "
Jul 09 17:44:31.366622 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.366582    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt5lh\" (UniqueName: \"kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh\") pod \"8d16d427-7996-478e-b49f-daabd93d61aa\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") "
Jul 09 17:44:31.366622 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.366623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util\") pod \"8d16d427-7996-478e-b49f-daabd93d61aa\" (UID: \"8d16d427-7996-478e-b49f-daabd93d61aa\") "
Jul 09 17:44:31.367211 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.367182    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle" (OuterVolumeSpecName: "bundle") pod "8d16d427-7996-478e-b49f-daabd93d61aa" (UID: "8d16d427-7996-478e-b49f-daabd93d61aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:44:31.368919 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.368891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh" (OuterVolumeSpecName: "kube-api-access-dt5lh") pod "8d16d427-7996-478e-b49f-daabd93d61aa" (UID: "8d16d427-7996-478e-b49f-daabd93d61aa"). InnerVolumeSpecName "kube-api-access-dt5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:44:31.370969 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.370941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util" (OuterVolumeSpecName: "util") pod "8d16d427-7996-478e-b49f-daabd93d61aa" (UID: "8d16d427-7996-478e-b49f-daabd93d61aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:44:31.467273 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.467242    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-bundle\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:44:31.467273 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.467267    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5lh\" (UniqueName: \"kubernetes.io/projected/8d16d427-7996-478e-b49f-daabd93d61aa-kube-api-access-dt5lh\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:44:31.467273 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:31.467278    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d16d427-7996-478e-b49f-daabd93d61aa-util\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:44:32.109090 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:32.109065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v"
Jul 09 17:44:32.109258 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:32.109066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589d9b2v" event={"ID":"8d16d427-7996-478e-b49f-daabd93d61aa","Type":"ContainerDied","Data":"3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408"}
Jul 09 17:44:32.109258 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:32.109179    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da78c38e37e33e69c77373b4c34a87207b035105f1b4c416de99c7110abb408"
Jul 09 17:44:42.742099 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.742065    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-szwb4"]
Jul 09 17:44:42.743003 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.742978    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="extract"
Jul 09 17:44:42.743003 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743004    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="extract"
Jul 09 17:44:42.743158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743018    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="util"
Jul 09 17:44:42.743158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743026    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="util"
Jul 09 17:44:42.743158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743054    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="pull"
Jul 09 17:44:42.743158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743062    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="pull"
Jul 09 17:44:42.743158 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.743143    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d16d427-7996-478e-b49f-daabd93d61aa" containerName="extract"
Jul 09 17:44:42.745381 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.745347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.748513 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.748489    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 09 17:44:42.748611 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.748540    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 09 17:44:42.748782 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.748767    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 09 17:44:42.748830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.748778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mgk2l\""
Jul 09 17:44:42.749956 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.749940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 09 17:44:42.750050 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.749974    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 09 17:44:42.754199 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.754171    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-szwb4"]
Jul 09 17:44:42.754779 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.754759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/277922da-8630-4852-99df-d7c4fff00a65-cabundle0\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.754947 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.754816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts464\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-kube-api-access-ts464\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.754947 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.754848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.855463 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.855434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.855643 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.855503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/277922da-8630-4852-99df-d7c4fff00a65-cabundle0\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.855643 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.855549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts464\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-kube-api-access-ts464\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.855643 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:42.855594    2577 secret.go:281] references non-existent secret key: ca.crt
Jul 09 17:44:42.855643 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:42.855614    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 09 17:44:42.855643 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:42.855630    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-szwb4: references non-existent secret key: ca.crt
Jul 09 17:44:42.855801 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:42.855697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates podName:277922da-8630-4852-99df-d7c4fff00a65 nodeName:}" failed. No retries permitted until 2026-07-09 17:44:43.355676225 +0000 UTC m=+296.958561676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates") pod "keda-operator-68bff59c-szwb4" (UID: "277922da-8630-4852-99df-d7c4fff00a65") : references non-existent secret key: ca.crt
Jul 09 17:44:42.856203 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.856186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/277922da-8630-4852-99df-d7c4fff00a65-cabundle0\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.866690 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.866659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts464\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-kube-api-access-ts464\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:42.947192 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.947158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"]
Jul 09 17:44:42.949915 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.949897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:42.952773 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.952755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 09 17:44:42.956585 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.956418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:42.956585 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.956531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwp5\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-kube-api-access-4qwp5\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:42.956710 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.956609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4de7f902-a152-4c06-a170-1fdfa8221a4a-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:42.959503 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:42.959479    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"]
Jul 09 17:44:43.057429 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.057312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwp5\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-kube-api-access-4qwp5\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.057429 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.057383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4de7f902-a152-4c06-a170-1fdfa8221a4a-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.057889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:43.058112    2577 secret.go:281] references non-existent secret key: tls.crt
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:43.058134    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.058135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4de7f902-a152-4c06-a170-1fdfa8221a4a-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:43.058154    2577 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:43.058175    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-85h8h: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 09 17:44:43.058292 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:44:43.058255    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates podName:4de7f902-a152-4c06-a170-1fdfa8221a4a nodeName:}" failed. No retries permitted until 2026-07-09 17:44:43.558229594 +0000 UTC m=+297.161115121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates") pod "keda-metrics-apiserver-559f75f947-85h8h" (UID: "4de7f902-a152-4c06-a170-1fdfa8221a4a") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 09 17:44:43.066748 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.066720    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwp5\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-kube-api-access-4qwp5\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.360749 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.360660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:43.363004 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.362986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/277922da-8630-4852-99df-d7c4fff00a65-certificates\") pod \"keda-operator-68bff59c-szwb4\" (UID: \"277922da-8630-4852-99df-d7c4fff00a65\") " pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:43.562440 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.562403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.564924 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.564904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4de7f902-a152-4c06-a170-1fdfa8221a4a-certificates\") pod \"keda-metrics-apiserver-559f75f947-85h8h\" (UID: \"4de7f902-a152-4c06-a170-1fdfa8221a4a\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.656646 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.656560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:43.783395 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.783351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-szwb4"]
Jul 09 17:44:43.785698 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:44:43.785674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277922da_8630_4852_99df_d7c4fff00a65.slice/crio-bc4ff665e724a9eb77b24aec97e6f6a4b76d40523e3e18308073614421eac3f5 WatchSource:0}: Error finding container bc4ff665e724a9eb77b24aec97e6f6a4b76d40523e3e18308073614421eac3f5: Status 404 returned error can't find the container with id bc4ff665e724a9eb77b24aec97e6f6a4b76d40523e3e18308073614421eac3f5
Jul 09 17:44:43.860752 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.860714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:43.997544 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:43.997508    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"]
Jul 09 17:44:43.999649 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:44:43.999621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de7f902_a152_4c06_a170_1fdfa8221a4a.slice/crio-66c2f8a865663a6ee0e9a44e5fac52e68b17fbe5e34cc2e81e508ff6a763f2d3 WatchSource:0}: Error finding container 66c2f8a865663a6ee0e9a44e5fac52e68b17fbe5e34cc2e81e508ff6a763f2d3: Status 404 returned error can't find the container with id 66c2f8a865663a6ee0e9a44e5fac52e68b17fbe5e34cc2e81e508ff6a763f2d3
Jul 09 17:44:44.157202 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:44.157170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h" event={"ID":"4de7f902-a152-4c06-a170-1fdfa8221a4a","Type":"ContainerStarted","Data":"66c2f8a865663a6ee0e9a44e5fac52e68b17fbe5e34cc2e81e508ff6a763f2d3"}
Jul 09 17:44:44.158283 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:44.158216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-szwb4" event={"ID":"277922da-8630-4852-99df-d7c4fff00a65","Type":"ContainerStarted","Data":"bc4ff665e724a9eb77b24aec97e6f6a4b76d40523e3e18308073614421eac3f5"}
Jul 09 17:44:48.281275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:48.281255    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:44:48.281738 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:48.281708    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:44:48.288005 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:48.287985    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 17:44:49.181288 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.181248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-szwb4" event={"ID":"277922da-8630-4852-99df-d7c4fff00a65","Type":"ContainerStarted","Data":"9ff95089f1ef09b92ec84ef88f1bd35c056c41be1928a8cb75ba3974b9df464e"}
Jul 09 17:44:49.181512 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.181313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:44:49.182768 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.182745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h" event={"ID":"4de7f902-a152-4c06-a170-1fdfa8221a4a","Type":"ContainerStarted","Data":"3562294a15303c086498ada2abab48213f88d26e18d3aaab4b81661d1cfe27a9"}
Jul 09 17:44:49.182874 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.182858    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:44:49.199852 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.199808    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-szwb4" podStartSLOduration=2.714226628 podStartE2EDuration="7.199794625s" totalImagesPullingTime="4.485567997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:49.198791495 +0000 UTC m=+302.801676954" watchObservedRunningTime="2026-07-09 17:44:49.199794625 +0000 UTC m=+302.802680080"
Jul 09 17:44:49.217724 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:44:49.217679    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h" podStartSLOduration=2.951376705 podStartE2EDuration="7.217665266s" totalImagesPullingTime="4.266288561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:49.216379139 +0000 UTC m=+302.819264596" watchObservedRunningTime="2026-07-09 17:44:49.217665266 +0000 UTC m=+302.820550722"
Jul 09 17:45:00.192190 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:00.192162    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-85h8h"
Jul 09 17:45:10.190299 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:10.190262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-szwb4"
Jul 09 17:45:58.198210 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.198176    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-htvdb"]
Jul 09 17:45:58.200656 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.200639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.207846 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.207830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pt4wf\""
Jul 09 17:45:58.208137 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.208119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 17:45:58.209194 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.209174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 17:45:58.209272 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.209179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 17:45:58.221259 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.221232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-htvdb"]
Jul 09 17:45:58.284677 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.284648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zh7z\" (UniqueName: \"kubernetes.io/projected/2fae0562-3b14-4677-9cb3-deaf67baa603-kube-api-access-9zh7z\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.284835 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.284701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fae0562-3b14-4677-9cb3-deaf67baa603-data\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.385818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.385780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fae0562-3b14-4677-9cb3-deaf67baa603-data\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.385994 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.385852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zh7z\" (UniqueName: \"kubernetes.io/projected/2fae0562-3b14-4677-9cb3-deaf67baa603-kube-api-access-9zh7z\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.386174 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.386153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fae0562-3b14-4677-9cb3-deaf67baa603-data\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.403232 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.403207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zh7z\" (UniqueName: \"kubernetes.io/projected/2fae0562-3b14-4677-9cb3-deaf67baa603-kube-api-access-9zh7z\") pod \"seaweedfs-64568bcd49-htvdb\" (UID: \"2fae0562-3b14-4677-9cb3-deaf67baa603\") " pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.510272 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.510188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:45:58.651275 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.651242    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-htvdb"]
Jul 09 17:45:58.654737 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:45:58.654708    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fae0562_3b14_4677_9cb3_deaf67baa603.slice/crio-72eb86f24e99eab5fe4cafde72dc3d2ae5f30dda579ee36fc14877855514159c WatchSource:0}: Error finding container 72eb86f24e99eab5fe4cafde72dc3d2ae5f30dda579ee36fc14877855514159c: Status 404 returned error can't find the container with id 72eb86f24e99eab5fe4cafde72dc3d2ae5f30dda579ee36fc14877855514159c
Jul 09 17:45:58.656391 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:58.656355    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:45:59.448673 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:45:59.448632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-htvdb" event={"ID":"2fae0562-3b14-4677-9cb3-deaf67baa603","Type":"ContainerStarted","Data":"72eb86f24e99eab5fe4cafde72dc3d2ae5f30dda579ee36fc14877855514159c"}
Jul 09 17:46:02.463903 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:46:02.463868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-htvdb" event={"ID":"2fae0562-3b14-4677-9cb3-deaf67baa603","Type":"ContainerStarted","Data":"09b8c99fe4842ccc8792a0961d188c616e6434951b1c4837f26138e5435412f7"}
Jul 09 17:46:02.464404 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:46:02.464003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:46:08.470518 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:46:08.470482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-htvdb"
Jul 09 17:46:08.491606 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:46:08.491555    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-htvdb" podStartSLOduration=7.135013913 podStartE2EDuration="10.49153968s" totalImagesPullingTime="3.356525767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:46:02.487412881 +0000 UTC m=+376.090298338" watchObservedRunningTime="2026-07-09 17:46:08.49153968 +0000 UTC m=+382.094425132"
Jul 09 17:47:12.610236 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.610154    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-2tmt4"]
Jul 09 17:47:12.612408 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.612391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.615873 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.615851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 17:47:12.615982 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.615883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qchhg\""
Jul 09 17:47:12.630441 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.630415    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m22w5"]
Jul 09 17:47:12.633072 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.633054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2tmt4"]
Jul 09 17:47:12.633188 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.633150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.636062 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.636035    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 17:47:12.636387 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.636372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-87g86\""
Jul 09 17:47:12.669993 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.669951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m22w5"]
Jul 09 17:47:12.683992 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.683966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4dvj\" (UniqueName: \"kubernetes.io/projected/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-kube-api-access-b4dvj\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.684121 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.684002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fdr\" (UniqueName: \"kubernetes.io/projected/905c8dc9-6851-4a0d-b66b-5b6da1131e74-kube-api-access-t4fdr\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.684121 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.684021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-tls-certs\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.684121 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.684076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/905c8dc9-6851-4a0d-b66b-5b6da1131e74-cert\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.784841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fdr\" (UniqueName: \"kubernetes.io/projected/905c8dc9-6851-4a0d-b66b-5b6da1131e74-kube-api-access-t4fdr\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.784889 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.784890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-tls-certs\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.785163 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.784911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/905c8dc9-6851-4a0d-b66b-5b6da1131e74-cert\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.785163 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.784977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4dvj\" (UniqueName: \"kubernetes.io/projected/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-kube-api-access-b4dvj\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.787328 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.787303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-tls-certs\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.787489 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.787469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/905c8dc9-6851-4a0d-b66b-5b6da1131e74-cert\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.797564 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.797535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4dvj\" (UniqueName: \"kubernetes.io/projected/0eb0906c-1980-4618-9b23-ff3f1d5e49dd-kube-api-access-b4dvj\") pod \"model-serving-api-fd65d7d6-2tmt4\" (UID: \"0eb0906c-1980-4618-9b23-ff3f1d5e49dd\") " pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.797829 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.797807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fdr\" (UniqueName: \"kubernetes.io/projected/905c8dc9-6851-4a0d-b66b-5b6da1131e74-kube-api-access-t4fdr\") pod \"odh-model-controller-84b54b7f97-m22w5\" (UID: \"905c8dc9-6851-4a0d-b66b-5b6da1131e74\") " pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:12.922924 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.922828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:12.942831 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:12.942800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:13.082693 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:13.082669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-2tmt4"]
Jul 09 17:47:13.085316 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:47:13.085288    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb0906c_1980_4618_9b23_ff3f1d5e49dd.slice/crio-b460fe7a31496e71fc1141817a1aa93817572c3bb9e9abe1c036dfc6b0966c37 WatchSource:0}: Error finding container b460fe7a31496e71fc1141817a1aa93817572c3bb9e9abe1c036dfc6b0966c37: Status 404 returned error can't find the container with id b460fe7a31496e71fc1141817a1aa93817572c3bb9e9abe1c036dfc6b0966c37
Jul 09 17:47:13.145870 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:13.145848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-m22w5"]
Jul 09 17:47:13.148351 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:47:13.148329    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905c8dc9_6851_4a0d_b66b_5b6da1131e74.slice/crio-15214738b6a09587a8b923d0b958a541f7154cfa2a35f0645385c3a456cc1881 WatchSource:0}: Error finding container 15214738b6a09587a8b923d0b958a541f7154cfa2a35f0645385c3a456cc1881: Status 404 returned error can't find the container with id 15214738b6a09587a8b923d0b958a541f7154cfa2a35f0645385c3a456cc1881
Jul 09 17:47:13.742019 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:13.741982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m22w5" event={"ID":"905c8dc9-6851-4a0d-b66b-5b6da1131e74","Type":"ContainerStarted","Data":"15214738b6a09587a8b923d0b958a541f7154cfa2a35f0645385c3a456cc1881"}
Jul 09 17:47:13.743234 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:13.743202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2tmt4" event={"ID":"0eb0906c-1980-4618-9b23-ff3f1d5e49dd","Type":"ContainerStarted","Data":"b460fe7a31496e71fc1141817a1aa93817572c3bb9e9abe1c036dfc6b0966c37"}
Jul 09 17:47:17.761995 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.761954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-m22w5" event={"ID":"905c8dc9-6851-4a0d-b66b-5b6da1131e74","Type":"ContainerStarted","Data":"3571dddc10a6c923abd461d203278d8fec88bc6f04dd34d12656122f3a0fbd32"}
Jul 09 17:47:17.762460 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.762082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:17.763534 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.763516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-2tmt4" event={"ID":"0eb0906c-1980-4618-9b23-ff3f1d5e49dd","Type":"ContainerStarted","Data":"c70cb546a2524964f1dfcb3fac76a4342b261499858bff56997e6f395ce7714c"}
Jul 09 17:47:17.763596 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.763566    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:47:17.782151 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.782108    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-m22w5" podStartSLOduration=1.793602699 podStartE2EDuration="5.782095084s" totalImagesPullingTime="3.988492385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:17.781629715 +0000 UTC m=+451.384515170" watchObservedRunningTime="2026-07-09 17:47:17.782095084 +0000 UTC m=+451.384980530"
Jul 09 17:47:17.803185 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:17.803141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-2tmt4" podStartSLOduration=1.7131425999999998 podStartE2EDuration="5.803126965s" totalImagesPullingTime="4.089984365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:17.802335533 +0000 UTC m=+451.405220990" watchObservedRunningTime="2026-07-09 17:47:17.803126965 +0000 UTC m=+451.406012476"
Jul 09 17:47:28.769754 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:28.769724    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-m22w5"
Jul 09 17:47:28.771400 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:47:28.771383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-2tmt4"
Jul 09 17:49:10.917334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.917292    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:49:10.920613 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.920595    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:10.924151 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.924126    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:49:10.925156 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.925133    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mspnj\""
Jul 09 17:49:10.925276 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.925174    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-654f2-kube-rbac-proxy-sar-config\""
Jul 09 17:49:10.925276 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.925173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-654f2-predictor-serving-cert\""
Jul 09 17:49:10.925605 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.925589    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:49:10.936515 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:10.936493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:49:11.085763 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.085711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.085920 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.085819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.085920 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.085867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hn8x\" (UniqueName: \"kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.186820 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.186780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.187016 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.186841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hn8x\" (UniqueName: \"kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.187016 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.186922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.187504 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.187478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.189292 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.189267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.197817 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.197787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hn8x\" (UniqueName: \"kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x\") pod \"success-200-isvc-654f2-predictor-59b774c954-qhsv8\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") " pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.231822 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.231794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:11.379211 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.379186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:49:11.381120 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:49:11.381091    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3322d87e_2ead_4833_a380_8f5c1c97a1a3.slice/crio-75d3847e0354aa80c6a93288d609bd66972079079ba504b8e6381b3cf0ea6c31 WatchSource:0}: Error finding container 75d3847e0354aa80c6a93288d609bd66972079079ba504b8e6381b3cf0ea6c31: Status 404 returned error can't find the container with id 75d3847e0354aa80c6a93288d609bd66972079079ba504b8e6381b3cf0ea6c31
Jul 09 17:49:11.621276 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.621191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:49:11.625703 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.625686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.628579 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.628559    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-654f2-predictor-serving-cert\""
Jul 09 17:49:11.628647 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.628559    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-654f2-kube-rbac-proxy-sar-config\""
Jul 09 17:49:11.640334 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.640314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:49:11.792345 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.792306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.792537 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.792408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq7v\" (UniqueName: \"kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.792537 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.792472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.893248 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.893146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.893248 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.893200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.893512 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.893259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq7v\" (UniqueName: \"kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.893933 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.893911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.895721 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.895701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.903179 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.903154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq7v\" (UniqueName: \"kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v\") pod \"error-404-isvc-654f2-predictor-75c6649bd7-rkpbb\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") " pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:11.936220 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:11.936187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:12.081181 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:12.081095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:49:12.085847 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:49:12.085805    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e1342b_9aef_469a_b803_1711f8549184.slice/crio-fd8d5a61c26f50965dd0cc488b28367ec5e1cee4564b4bd8cb9af7051252ca33 WatchSource:0}: Error finding container fd8d5a61c26f50965dd0cc488b28367ec5e1cee4564b4bd8cb9af7051252ca33: Status 404 returned error can't find the container with id fd8d5a61c26f50965dd0cc488b28367ec5e1cee4564b4bd8cb9af7051252ca33
Jul 09 17:49:12.219522 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:12.218489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerStarted","Data":"fd8d5a61c26f50965dd0cc488b28367ec5e1cee4564b4bd8cb9af7051252ca33"}
Jul 09 17:49:12.220416 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:12.220389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerStarted","Data":"75d3847e0354aa80c6a93288d609bd66972079079ba504b8e6381b3cf0ea6c31"}
Jul 09 17:49:28.327139 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:28.327095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerStarted","Data":"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"}
Jul 09 17:49:28.329255 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:28.329176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerStarted","Data":"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"}
Jul 09 17:49:31.347202 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.347160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerStarted","Data":"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"}
Jul 09 17:49:31.347700 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.347241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:31.347700 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.347274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:31.348738 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.348700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:49:31.349066 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.349045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerStarted","Data":"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"}
Jul 09 17:49:31.349175 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.349103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:31.349175 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.349137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:31.350296 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.350270    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:49:31.397165 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.397107    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podStartSLOduration=2.123656105 podStartE2EDuration="20.39709044s" totalImagesPullingTime="18.273434335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:49:31.38577646 +0000 UTC m=+584.988661942" watchObservedRunningTime="2026-07-09 17:49:31.39709044 +0000 UTC m=+584.999975894"
Jul 09 17:49:31.421952 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:31.421901    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podStartSLOduration=2.45787006 podStartE2EDuration="21.421885026s" totalImagesPullingTime="18.964014966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:49:31.420963743 +0000 UTC m=+585.023849199" watchObservedRunningTime="2026-07-09 17:49:31.421885026 +0000 UTC m=+585.024770481"
Jul 09 17:49:32.353963 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:32.353907    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:49:32.353963 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:32.353962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:49:37.358263 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:37.358220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:49:37.358715 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:37.358652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:49:37.358715 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:37.358652    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:49:37.359077 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:37.359055    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:49:47.358779 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:47.358729    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:49:47.359273 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:47.359042    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:49:48.313758 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:48.313646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:49:48.313758 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:48.313652    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:49:57.358966 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:57.358927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:49:57.359337 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:49:57.359060    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:50:07.358726 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:07.358687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 09 17:50:07.359178 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:07.359137    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 09 17:50:17.359830 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:17.359749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:50:17.360227 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:17.360209    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:50:41.139040 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.139010    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:50:41.139516 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.139274    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container" containerID="cri-o://3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7" gracePeriod=30
Jul 09 17:50:41.139516 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.139325    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kube-rbac-proxy" containerID="cri-o://e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5" gracePeriod=30
Jul 09 17:50:41.195755 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.195728    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:50:41.195982 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.195955    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container" containerID="cri-o://87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e" gracePeriod=30
Jul 09 17:50:41.196047 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.195980    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kube-rbac-proxy" containerID="cri-o://404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c" gracePeriod=30
Jul 09 17:50:41.310008 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.309981    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 17:50:41.313298 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.313279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.316398 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.316380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-785cd-predictor-serving-cert\""
Jul 09 17:50:41.316490 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.316399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-785cd-kube-rbac-proxy-sar-config\""
Jul 09 17:50:41.325594 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.325574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 17:50:41.376818 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.376788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.376978 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.376832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.376978 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.376878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgz7\" (UniqueName: \"kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.431936 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.431909    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 17:50:41.435643 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.435627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.438845 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.438826    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-785cd-kube-rbac-proxy-sar-config\""
Jul 09 17:50:41.439315 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.439300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-785cd-predictor-serving-cert\""
Jul 09 17:50:41.447016 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.446998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 17:50:41.477402 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgz7\" (UniqueName: \"kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.477507 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.477507 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.477507 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64t57\" (UniqueName: \"kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.477602 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.477602 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.477561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.477676 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:41.477657    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-785cd-predictor-serving-cert: secret "success-200-isvc-785cd-predictor-serving-cert" not found
Jul 09 17:50:41.477741 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:41.477730    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls podName:e2d67e88-66fd-44f1-96fa-0849e42332ac nodeName:}" failed. No retries permitted until 2026-07-09 17:50:41.977702951 +0000 UTC m=+655.580588386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls") pod "success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" (UID: "e2d67e88-66fd-44f1-96fa-0849e42332ac") : secret "success-200-isvc-785cd-predictor-serving-cert" not found
Jul 09 17:50:41.478178 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.478163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.489544 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.489526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgz7\" (UniqueName: \"kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.578438 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.578405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.578577 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.578443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64t57\" (UniqueName: \"kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.578577 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.578551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.578686 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:41.578651    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-785cd-predictor-serving-cert: secret "error-404-isvc-785cd-predictor-serving-cert" not found
Jul 09 17:50:41.578739 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:41.578723    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls podName:f243305b-4440-4c3d-bf34-ae1bcb2c2ec5 nodeName:}" failed. No retries permitted until 2026-07-09 17:50:42.078696551 +0000 UTC m=+655.681581987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls") pod "error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" (UID: "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5") : secret "error-404-isvc-785cd-predictor-serving-cert" not found
Jul 09 17:50:41.579142 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.579125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.587644 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.587619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64t57\" (UniqueName: \"kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:41.638780 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.638754    2577 generic.go:356] "Generic (PLEG): container finished" podID="97e1342b-9aef-469a-b803-1711f8549184" containerID="e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5" exitCode=2
Jul 09 17:50:41.638888 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.638826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerDied","Data":"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"}
Jul 09 17:50:41.640628 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.640609    2577 generic.go:356] "Generic (PLEG): container finished" podID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerID="404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c" exitCode=2
Jul 09 17:50:41.640716 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.640679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerDied","Data":"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"}
Jul 09 17:50:41.983789 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.983745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:41.986407 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:41.986350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") pod \"success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") " pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:42.084955 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.084915    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:42.087432 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.087404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") pod \"error-404-isvc-785cd-predictor-c6659f9bc-ptmcr\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") " pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:42.223435 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.223406    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:42.346394 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.346342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:42.354665 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.354548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 09 17:50:42.354844 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.354807    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 09 17:50:42.357555 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.357532    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 17:50:42.359200 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:50:42.359178    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d67e88_66fd_44f1_96fa_0849e42332ac.slice/crio-8599a86564616962fb2cb1082b7e3370b01419b0290c25b9753275f6fb9082ef WatchSource:0}: Error finding container 8599a86564616962fb2cb1082b7e3370b01419b0290c25b9753275f6fb9082ef: Status 404 returned error can't find the container with id 8599a86564616962fb2cb1082b7e3370b01419b0290c25b9753275f6fb9082ef
Jul 09 17:50:42.487649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.487621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 17:50:42.489925 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:50:42.489897    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf243305b_4440_4c3d_bf34_ae1bcb2c2ec5.slice/crio-6805ac0711b0e0fac09c8c6810ba9ff779a4c2dfa56ee1e59878a7822e074b9d WatchSource:0}: Error finding container 6805ac0711b0e0fac09c8c6810ba9ff779a4c2dfa56ee1e59878a7822e074b9d: Status 404 returned error can't find the container with id 6805ac0711b0e0fac09c8c6810ba9ff779a4c2dfa56ee1e59878a7822e074b9d
Jul 09 17:50:42.648914 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.648886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerStarted","Data":"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"}
Jul 09 17:50:42.649039 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.648953    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:42.649039 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.648978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerStarted","Data":"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"}
Jul 09 17:50:42.649039 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.649004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerStarted","Data":"8599a86564616962fb2cb1082b7e3370b01419b0290c25b9753275f6fb9082ef"}
Jul 09 17:50:42.649039 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.649033    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:42.650532 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650509    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:50:42.650754 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerStarted","Data":"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"}
Jul 09 17:50:42.650841 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerStarted","Data":"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"}
Jul 09 17:50:42.650841 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:42.650841 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerStarted","Data":"6805ac0711b0e0fac09c8c6810ba9ff779a4c2dfa56ee1e59878a7822e074b9d"}
Jul 09 17:50:42.650975 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.650865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:42.651957 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.651933    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:50:42.669883 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.669840    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podStartSLOduration=1.669826317 podStartE2EDuration="1.669826317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:42.668418082 +0000 UTC m=+656.271303538" watchObservedRunningTime="2026-07-09 17:50:42.669826317 +0000 UTC m=+656.272711774"
Jul 09 17:50:42.689111 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:42.689052    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podStartSLOduration=1.689031336 podStartE2EDuration="1.689031336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:42.688458738 +0000 UTC m=+656.291344215" watchObservedRunningTime="2026-07-09 17:50:42.689031336 +0000 UTC m=+656.291916799"
Jul 09 17:50:43.655402 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:43.655345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:50:43.655843 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:43.655496    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:50:44.304508 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.304488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:50:44.351968 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.351914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:50:44.407514 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hn8x\" (UniqueName: \"kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x\") pod \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") "
Jul 09 17:50:44.407514 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config\") pod \"97e1342b-9aef-469a-b803-1711f8549184\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") "
Jul 09 17:50:44.407719 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls\") pod \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") "
Jul 09 17:50:44.407719 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config\") pod \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\" (UID: \"3322d87e-2ead-4833-a380-8f5c1c97a1a3\") "
Jul 09 17:50:44.407719 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jq7v\" (UniqueName: \"kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v\") pod \"97e1342b-9aef-469a-b803-1711f8549184\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") "
Jul 09 17:50:44.407719 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls\") pod \"97e1342b-9aef-469a-b803-1711f8549184\" (UID: \"97e1342b-9aef-469a-b803-1711f8549184\") "
Jul 09 17:50:44.407991 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97e1342b-9aef-469a-b803-1711f8549184" (UID: "97e1342b-9aef-469a-b803-1711f8549184"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:50:44.408070 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.407967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3322d87e-2ead-4833-a380-8f5c1c97a1a3" (UID: "3322d87e-2ead-4833-a380-8f5c1c97a1a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:50:44.409779 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.409744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97e1342b-9aef-469a-b803-1711f8549184" (UID: "97e1342b-9aef-469a-b803-1711f8549184"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:50:44.409879 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.409825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x" (OuterVolumeSpecName: "kube-api-access-9hn8x") pod "3322d87e-2ead-4833-a380-8f5c1c97a1a3" (UID: "3322d87e-2ead-4833-a380-8f5c1c97a1a3"). InnerVolumeSpecName "kube-api-access-9hn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:50:44.410038 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.410018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3322d87e-2ead-4833-a380-8f5c1c97a1a3" (UID: "3322d87e-2ead-4833-a380-8f5c1c97a1a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:50:44.410093 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.410021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v" (OuterVolumeSpecName: "kube-api-access-6jq7v") pod "97e1342b-9aef-469a-b803-1711f8549184" (UID: "97e1342b-9aef-469a-b803-1711f8549184"). InnerVolumeSpecName "kube-api-access-6jq7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:50:44.509026 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509002    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97e1342b-9aef-469a-b803-1711f8549184-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.509026 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509023    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hn8x\" (UniqueName: \"kubernetes.io/projected/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-api-access-9hn8x\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.509161 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509033    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97e1342b-9aef-469a-b803-1711f8549184-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.509161 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509042    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3322d87e-2ead-4833-a380-8f5c1c97a1a3-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.509161 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509051    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3322d87e-2ead-4833-a380-8f5c1c97a1a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.509161 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.509060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jq7v\" (UniqueName: \"kubernetes.io/projected/97e1342b-9aef-469a-b803-1711f8549184-kube-api-access-6jq7v\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:50:44.660925 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.660835    2577 generic.go:356] "Generic (PLEG): container finished" podID="97e1342b-9aef-469a-b803-1711f8549184" containerID="3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7" exitCode=0
Jul 09 17:50:44.660925 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.660889    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"
Jul 09 17:50:44.661409 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.660925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerDied","Data":"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"}
Jul 09 17:50:44.661409 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.660992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb" event={"ID":"97e1342b-9aef-469a-b803-1711f8549184","Type":"ContainerDied","Data":"fd8d5a61c26f50965dd0cc488b28367ec5e1cee4564b4bd8cb9af7051252ca33"}
Jul 09 17:50:44.661409 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.661018    2577 scope.go:117] "RemoveContainer" containerID="e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"
Jul 09 17:50:44.662560 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.662527    2577 generic.go:356] "Generic (PLEG): container finished" podID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerID="87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e" exitCode=0
Jul 09 17:50:44.662660 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.662578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"
Jul 09 17:50:44.662660 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.662604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerDied","Data":"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"}
Jul 09 17:50:44.662764 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.662659    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8" event={"ID":"3322d87e-2ead-4833-a380-8f5c1c97a1a3","Type":"ContainerDied","Data":"75d3847e0354aa80c6a93288d609bd66972079079ba504b8e6381b3cf0ea6c31"}
Jul 09 17:50:44.669908 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.669878    2577 scope.go:117] "RemoveContainer" containerID="3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"
Jul 09 17:50:44.677394 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.677374    2577 scope.go:117] "RemoveContainer" containerID="e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"
Jul 09 17:50:44.677636 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:44.677617    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5\": container with ID starting with e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5 not found: ID does not exist" containerID="e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"
Jul 09 17:50:44.677693 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.677647    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5"} err="failed to get container status \"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5\": rpc error: code = NotFound desc = could not find container \"e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5\": container with ID starting with e76295d78854d9eab1b49ea922904b6a32c357c2d1bd6a73b8ba7a7ea87000c5 not found: ID does not exist"
Jul 09 17:50:44.677693 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.677673    2577 scope.go:117] "RemoveContainer" containerID="3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"
Jul 09 17:50:44.677906 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:44.677891    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7\": container with ID starting with 3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7 not found: ID does not exist" containerID="3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"
Jul 09 17:50:44.677950 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.677909    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7"} err="failed to get container status \"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7\": rpc error: code = NotFound desc = could not find container \"3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7\": container with ID starting with 3bdd1ebe14356b18faa05cd489798c616752cddddc180b7b3d4cc19b1f999dd7 not found: ID does not exist"
Jul 09 17:50:44.677950 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.677922    2577 scope.go:117] "RemoveContainer" containerID="404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"
Jul 09 17:50:44.685030 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.685006    2577 scope.go:117] "RemoveContainer" containerID="87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"
Jul 09 17:50:44.687382 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.687335    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:50:44.691541 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.691520    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-654f2-predictor-59b774c954-qhsv8"]
Jul 09 17:50:44.693921 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.693904    2577 scope.go:117] "RemoveContainer" containerID="404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"
Jul 09 17:50:44.694190 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:44.694166    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c\": container with ID starting with 404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c not found: ID does not exist" containerID="404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"
Jul 09 17:50:44.694262 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.694196    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c"} err="failed to get container status \"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c\": rpc error: code = NotFound desc = could not find container \"404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c\": container with ID starting with 404e939c546eae6bbf7d629b286a731bcfcca3d84ee2cd3dd88af06c6705f12c not found: ID does not exist"
Jul 09 17:50:44.694262 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.694219    2577 scope.go:117] "RemoveContainer" containerID="87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"
Jul 09 17:50:44.694499 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:50:44.694482    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e\": container with ID starting with 87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e not found: ID does not exist" containerID="87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"
Jul 09 17:50:44.694587 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.694506    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e"} err="failed to get container status \"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e\": rpc error: code = NotFound desc = could not find container \"87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e\": container with ID starting with 87112ee263f41ac3c87af0233530328227096bf03262c3bd2e792b387ac0041e not found: ID does not exist"
Jul 09 17:50:44.703750 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.703729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:50:44.708057 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.708039    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-654f2-predictor-75c6649bd7-rkpbb"]
Jul 09 17:50:44.966237 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.966198    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" path="/var/lib/kubelet/pods/3322d87e-2ead-4833-a380-8f5c1c97a1a3/volumes"
Jul 09 17:50:44.966716 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:44.966695    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e1342b-9aef-469a-b803-1711f8549184" path="/var/lib/kubelet/pods/97e1342b-9aef-469a-b803-1711f8549184/volumes"
Jul 09 17:50:48.660603 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:48.660577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:50:48.661053 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:48.660651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:50:48.661053 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:48.661032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:50:48.661163 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:48.661047    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:50:58.661512 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:58.661469    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:50:58.661913 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:50:58.661491    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:51:08.661208 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:08.661159    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:51:08.661208 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:08.661159    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:51:18.661008 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:18.660967    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:51:18.661488 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:18.660968    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:51:28.662104 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:28.662073    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:51:28.662565 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:28.662174    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:51:41.289985 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.289911    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290293    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290314    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290320    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290346    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container"
Jul 09 17:51:41.290356 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290351    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container"
Jul 09 17:51:41.290649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290435    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kube-rbac-proxy"
Jul 09 17:51:41.290649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290452    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3322d87e-2ead-4833-a380-8f5c1c97a1a3" containerName="kserve-container"
Jul 09 17:51:41.290649 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.290458    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e1342b-9aef-469a-b803-1711f8549184" containerName="kserve-container"
Jul 09 17:51:41.293435 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.293419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.298666 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.298640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-71ac7-predictor-serving-cert\""
Jul 09 17:51:41.298782 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.298672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-71ac7-kube-rbac-proxy-sar-config\""
Jul 09 17:51:41.324190 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.324170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 17:51:41.371541 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.371508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.371692 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.371671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.371761 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.371711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w7xm\" (UniqueName: \"kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.473177 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.473146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.473342 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.473186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w7xm\" (UniqueName: \"kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.473342 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.473310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.473952 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.473927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.475679 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.475661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.510107 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.510084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w7xm\" (UniqueName: \"kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm\") pod \"success-200-isvc-71ac7-predictor-6899d48d94-czfzn\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") " pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.545272 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.545210    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 17:51:41.548891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.548875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.552192 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.552172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-71ac7-predictor-serving-cert\""
Jul 09 17:51:41.552268 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.552189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-71ac7-kube-rbac-proxy-sar-config\""
Jul 09 17:51:41.562174 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.562153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 17:51:41.603266 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.603236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.674948 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.674922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.675107 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.674975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swr69\" (UniqueName: \"kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.675107 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.675037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.738971 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.738941    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 17:51:41.740630 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:51:41.740602    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a229921_0ad5_4fc3_947f_4b0c2f8f029d.slice/crio-74bcbf435f0516d0e69dd2acfac286d078c66b1f07bfa41789b1b5ed1de5aad1 WatchSource:0}: Error finding container 74bcbf435f0516d0e69dd2acfac286d078c66b1f07bfa41789b1b5ed1de5aad1: Status 404 returned error can't find the container with id 74bcbf435f0516d0e69dd2acfac286d078c66b1f07bfa41789b1b5ed1de5aad1
Jul 09 17:51:41.776374 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.776344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.776469 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.776413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swr69\" (UniqueName: \"kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.776469 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.776450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.777114 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.777091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.778810 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.778788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.785008 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.784987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swr69\" (UniqueName: \"kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69\") pod \"error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") " pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.858963 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.858942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:41.907783 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.907703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerStarted","Data":"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"}
Jul 09 17:51:41.907783 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.907767    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerStarted","Data":"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"}
Jul 09 17:51:41.907941 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.907836    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerStarted","Data":"74bcbf435f0516d0e69dd2acfac286d078c66b1f07bfa41789b1b5ed1de5aad1"}
Jul 09 17:51:41.908114 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.908085    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.908177 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.908144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:41.909695 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.909644    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:51:41.929741 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:41.929672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podStartSLOduration=0.929649718 podStartE2EDuration="929.649718ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:41.926498981 +0000 UTC m=+715.529384463" watchObservedRunningTime="2026-07-09 17:51:41.929649718 +0000 UTC m=+715.532535177"
Jul 09 17:51:42.002968 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.002939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 17:51:42.004405 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:51:42.004379    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c22b1bd_0f20_4bd8_8323_1467b767b87d.slice/crio-4240e682553201cf03f4e4c0639a585fed3789cc07d80d8f88cafb5a64034129 WatchSource:0}: Error finding container 4240e682553201cf03f4e4c0639a585fed3789cc07d80d8f88cafb5a64034129: Status 404 returned error can't find the container with id 4240e682553201cf03f4e4c0639a585fed3789cc07d80d8f88cafb5a64034129
Jul 09 17:51:42.914076 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerStarted","Data":"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"}
Jul 09 17:51:42.914572 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerStarted","Data":"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"}
Jul 09 17:51:42.914572 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerStarted","Data":"4240e682553201cf03f4e4c0639a585fed3789cc07d80d8f88cafb5a64034129"}
Jul 09 17:51:42.914572 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:42.914572 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:42.914572 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.914550    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:51:42.915798 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.915773    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:51:42.936144 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:42.936103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podStartSLOduration=1.936088468 podStartE2EDuration="1.936088468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:42.933079479 +0000 UTC m=+716.535964938" watchObservedRunningTime="2026-07-09 17:51:42.936088468 +0000 UTC m=+716.538973923"
Jul 09 17:51:43.919089 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:43.919052    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:51:47.919457 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:47.919429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:51:47.919902 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:47.919876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:51:48.924285 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:48.924259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:51:48.924736 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:48.924709    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:51:57.920206 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:57.920165    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:51:58.925252 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:51:58.925212    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:52:07.920601 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:07.920560    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:52:08.925086 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:08.925046    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:52:17.920166 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:17.920129    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 17:52:18.925741 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:18.925689    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 17:52:27.920528 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:27.920493    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 17:52:28.925523 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:52:28.925495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 17:54:48.341065 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:54:48.340964    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:54:48.347012 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:54:48.343067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:59:48.368037 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:48.367936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:59:48.371905 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:48.371788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 17:59:56.042611 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.042570    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 17:59:56.043054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.042820    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" containerID="cri-o://75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a" gracePeriod=30
Jul 09 17:59:56.043054 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.042853    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kube-rbac-proxy" containerID="cri-o://cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf" gracePeriod=30
Jul 09 17:59:56.098253 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.098223    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 17:59:56.101906 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.101886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.106928 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.106902    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5bd2e-predictor-serving-cert\""
Jul 09 17:59:56.107061 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.106973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5bd2e-kube-rbac-proxy-sar-config\""
Jul 09 17:59:56.115544 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.115519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 17:59:56.178317 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.178282    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 17:59:56.181891 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.181873    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.184892 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.184871    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5bd2e-predictor-serving-cert\""
Jul 09 17:59:56.185012 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.184936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5bd2e-kube-rbac-proxy-sar-config\""
Jul 09 17:59:56.194291 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.194268    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 17:59:56.201443 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.201550 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwr7\" (UniqueName: \"kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.201550 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.201636 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.201694 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54j4\" (UniqueName: \"kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.201733 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.201722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.224643 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.224611    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 17:59:56.224866 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.224838    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" containerID="cri-o://f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99" gracePeriod=30
Jul 09 17:59:56.225068 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.224880    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kube-rbac-proxy" containerID="cri-o://f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9" gracePeriod=30
Jul 09 17:59:56.303118 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.303118 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54j4\" (UniqueName: \"kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.303118 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.303425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.303425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwr7\" (UniqueName: \"kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.303425 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.303425 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:59:56.303304    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-serving-cert: secret "error-404-isvc-5bd2e-predictor-serving-cert" not found
Jul 09 17:59:56.303425 ip-10-0-136-100 kubenswrapper[2577]: E0709 17:59:56.303381    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls podName:4277b038-4f02-471b-8133-e36bf4336e49 nodeName:}" failed. No retries permitted until 2026-07-09 17:59:56.803343942 +0000 UTC m=+1210.406229393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls") pod "error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" (UID: "4277b038-4f02-471b-8133-e36bf4336e49") : secret "error-404-isvc-5bd2e-predictor-serving-cert" not found
Jul 09 17:59:56.303882 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.303987 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.303916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.305844 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.305822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.312349 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.312325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54j4\" (UniqueName: \"kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4\") pod \"success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") " pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.312628 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.312605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwr7\" (UniqueName: \"kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.417516 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.417478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:56.554313 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.554284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 17:59:56.556214 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:59:56.556190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6f781e_b38d_4923_90dc_b95195ad9adb.slice/crio-f6b641336d38dd60ab10a31426043b2af199641a2af48eeb0a68b05c7a93cf88 WatchSource:0}: Error finding container f6b641336d38dd60ab10a31426043b2af199641a2af48eeb0a68b05c7a93cf88: Status 404 returned error can't find the container with id f6b641336d38dd60ab10a31426043b2af199641a2af48eeb0a68b05c7a93cf88
Jul 09 17:59:56.808416 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.808311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:56.810660 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:56.810640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") pod \"error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") " pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:57.092790 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.092694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:57.118264 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.118231    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerID="cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf" exitCode=2
Jul 09 17:59:57.118437 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.118294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerDied","Data":"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"}
Jul 09 17:59:57.120184 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.120159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerStarted","Data":"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"}
Jul 09 17:59:57.120279 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.120209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerStarted","Data":"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"}
Jul 09 17:59:57.120279 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.120241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:57.120279 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.120264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerStarted","Data":"f6b641336d38dd60ab10a31426043b2af199641a2af48eeb0a68b05c7a93cf88"}
Jul 09 17:59:57.120482 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.120283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 17:59:57.122256 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.122225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 17:59:57.122429 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.122408    2577 generic.go:356] "Generic (PLEG): container finished" podID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerID="f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9" exitCode=2
Jul 09 17:59:57.122495 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.122469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerDied","Data":"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"}
Jul 09 17:59:57.141834 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.141794    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podStartSLOduration=1.141778078 podStartE2EDuration="1.141778078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:59:57.140341623 +0000 UTC m=+1210.743227080" watchObservedRunningTime="2026-07-09 17:59:57.141778078 +0000 UTC m=+1210.744663533"
Jul 09 17:59:57.234560 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:57.234536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 17:59:57.236289 ip-10-0-136-100 kubenswrapper[2577]: W0709 17:59:57.236252    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4277b038_4f02_471b_8133_e36bf4336e49.slice/crio-87fdc7fa0b4773a13e8eda9f039cb280b3a966b44c21b44357296d3fe8acade7 WatchSource:0}: Error finding container 87fdc7fa0b4773a13e8eda9f039cb280b3a966b44c21b44357296d3fe8acade7: Status 404 returned error can't find the container with id 87fdc7fa0b4773a13e8eda9f039cb280b3a966b44c21b44357296d3fe8acade7
Jul 09 17:59:58.130228 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerStarted","Data":"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"}
Jul 09 17:59:58.130696 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerStarted","Data":"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"}
Jul 09 17:59:58.130696 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130486    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:58.130696 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerStarted","Data":"87fdc7fa0b4773a13e8eda9f039cb280b3a966b44c21b44357296d3fe8acade7"}
Jul 09 17:59:58.130696 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130486    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 17:59:58.130696 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.130592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 17:59:58.131854 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.131834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 17:59:58.151893 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.151847    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podStartSLOduration=2.151833566 podStartE2EDuration="2.151833566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:59:58.149206024 +0000 UTC m=+1211.752091503" watchObservedRunningTime="2026-07-09 17:59:58.151833566 +0000 UTC m=+1211.754719021"
Jul 09 17:59:58.655803 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.655764    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 09 17:59:58.655980 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.655806    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 09 17:59:58.661235 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.661214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 09 17:59:58.661235 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:58.661222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 09 17:59:59.135407 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.135353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 17:59:59.368083 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.368059    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 17:59:59.433528 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.433506    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config\") pod \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") "
Jul 09 17:59:59.433684 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.433593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64t57\" (UniqueName: \"kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57\") pod \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") "
Jul 09 17:59:59.433730 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.433695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") pod \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\" (UID: \"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5\") "
Jul 09 17:59:59.433968 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.433947    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" (UID: "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:59:59.435680 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.435654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" (UID: "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:59:59.435776 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.435760    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57" (OuterVolumeSpecName: "kube-api-access-64t57") pod "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" (UID: "f243305b-4440-4c3d-bf34-ae1bcb2c2ec5"). InnerVolumeSpecName "kube-api-access-64t57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:59:59.534333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.534300    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:59:59.534333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.534329    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:59:59.534333 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.534340    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64t57\" (UniqueName: \"kubernetes.io/projected/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5-kube-api-access-64t57\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:59:59.685385 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.685290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 17:59:59.735998 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.735969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config\") pod \"e2d67e88-66fd-44f1-96fa-0849e42332ac\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") "
Jul 09 17:59:59.736173 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.736078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") pod \"e2d67e88-66fd-44f1-96fa-0849e42332ac\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") "
Jul 09 17:59:59.736173 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.736113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsgz7\" (UniqueName: \"kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7\") pod \"e2d67e88-66fd-44f1-96fa-0849e42332ac\" (UID: \"e2d67e88-66fd-44f1-96fa-0849e42332ac\") "
Jul 09 17:59:59.736391 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.736344    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2d67e88-66fd-44f1-96fa-0849e42332ac" (UID: "e2d67e88-66fd-44f1-96fa-0849e42332ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:59:59.738283 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.738252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7" (OuterVolumeSpecName: "kube-api-access-jsgz7") pod "e2d67e88-66fd-44f1-96fa-0849e42332ac" (UID: "e2d67e88-66fd-44f1-96fa-0849e42332ac"). InnerVolumeSpecName "kube-api-access-jsgz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:59:59.738636 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.738616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2d67e88-66fd-44f1-96fa-0849e42332ac" (UID: "e2d67e88-66fd-44f1-96fa-0849e42332ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:59:59.836854 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.836821    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2d67e88-66fd-44f1-96fa-0849e42332ac-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:59:59.836854 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.836849    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgz7\" (UniqueName: \"kubernetes.io/projected/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-api-access-jsgz7\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 17:59:59.836854 ip-10-0-136-100 kubenswrapper[2577]: I0709 17:59:59.836858    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2d67e88-66fd-44f1-96fa-0849e42332ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:00.142133 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.142031    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerID="75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a" exitCode=0
Jul 09 18:00:00.142133 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.142088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"
Jul 09 18:00:00.142133 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.142120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerDied","Data":"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"}
Jul 09 18:00:00.142695 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.142189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk" event={"ID":"e2d67e88-66fd-44f1-96fa-0849e42332ac","Type":"ContainerDied","Data":"8599a86564616962fb2cb1082b7e3370b01419b0290c25b9753275f6fb9082ef"}
Jul 09 18:00:00.142695 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.142230    2577 scope.go:117] "RemoveContainer" containerID="cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"
Jul 09 18:00:00.143896 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.143871    2577 generic.go:356] "Generic (PLEG): container finished" podID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerID="f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99" exitCode=0
Jul 09 18:00:00.143993 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.143912    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"
Jul 09 18:00:00.143993 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.143953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerDied","Data":"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"}
Jul 09 18:00:00.144126 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.144014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr" event={"ID":"f243305b-4440-4c3d-bf34-ae1bcb2c2ec5","Type":"ContainerDied","Data":"6805ac0711b0e0fac09c8c6810ba9ff779a4c2dfa56ee1e59878a7822e074b9d"}
Jul 09 18:00:00.152019 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.151891    2577 scope.go:117] "RemoveContainer" containerID="75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"
Jul 09 18:00:00.159806 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.159791    2577 scope.go:117] "RemoveContainer" containerID="cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"
Jul 09 18:00:00.160044 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:00.160026    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf\": container with ID starting with cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf not found: ID does not exist" containerID="cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"
Jul 09 18:00:00.160082 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.160057    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf"} err="failed to get container status \"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf\": rpc error: code = NotFound desc = could not find container \"cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf\": container with ID starting with cebe197913b0464373af93a1c148b3043261b331343cf0fc1780b4fdc5c70ecf not found: ID does not exist"
Jul 09 18:00:00.160082 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.160079    2577 scope.go:117] "RemoveContainer" containerID="75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"
Jul 09 18:00:00.160282 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:00.160267    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a\": container with ID starting with 75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a not found: ID does not exist" containerID="75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"
Jul 09 18:00:00.160323 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.160288    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a"} err="failed to get container status \"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a\": rpc error: code = NotFound desc = could not find container \"75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a\": container with ID starting with 75199cf588c065b7bb001dde0252198869363400b68fbe9bd34efd1e2a046d6a not found: ID does not exist"
Jul 09 18:00:00.160323 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.160302    2577 scope.go:117] "RemoveContainer" containerID="f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"
Jul 09 18:00:00.167976 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.167890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 18:00:00.168034 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.167996    2577 scope.go:117] "RemoveContainer" containerID="f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"
Jul 09 18:00:00.172888 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.172865    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-785cd-predictor-7cbb69cdbb-tmptk"]
Jul 09 18:00:00.176654 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.176635    2577 scope.go:117] "RemoveContainer" containerID="f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"
Jul 09 18:00:00.176909 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:00.176894    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9\": container with ID starting with f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9 not found: ID does not exist" containerID="f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"
Jul 09 18:00:00.176954 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.176916    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9"} err="failed to get container status \"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9\": rpc error: code = NotFound desc = could not find container \"f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9\": container with ID starting with f74e4a840b6ef92f0fb502ac9db6da432e810195457992fc9b21fed707dd76e9 not found: ID does not exist"
Jul 09 18:00:00.176954 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.176935    2577 scope.go:117] "RemoveContainer" containerID="f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"
Jul 09 18:00:00.177199 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:00.177179    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99\": container with ID starting with f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99 not found: ID does not exist" containerID="f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"
Jul 09 18:00:00.177253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.177204    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99"} err="failed to get container status \"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99\": rpc error: code = NotFound desc = could not find container \"f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99\": container with ID starting with f4183b50cd32d69b30fe2bf235592a0f7b3d451931f074c7b44c404545bdab99 not found: ID does not exist"
Jul 09 18:00:00.186690 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.186661    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 18:00:00.192124 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.192094    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-785cd-predictor-c6659f9bc-ptmcr"]
Jul 09 18:00:00.967159 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.967128    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" path="/var/lib/kubelet/pods/e2d67e88-66fd-44f1-96fa-0849e42332ac/volumes"
Jul 09 18:00:00.967561 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:00.967547    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" path="/var/lib/kubelet/pods/f243305b-4440-4c3d-bf34-ae1bcb2c2ec5/volumes"
Jul 09 18:00:03.134638 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:03.134607    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 18:00:03.135048 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:03.135023    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:00:04.140554 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:04.140524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 18:00:04.140987 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:04.140963    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:00:13.136043 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:13.136004    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:00:14.141837 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:14.141801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:00:23.135093 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:23.135056    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:00:24.141556 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:24.141521    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:00:33.135155 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:33.135117    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 09 18:00:34.141062 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:34.141027    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 09 18:00:43.136541 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:43.136467    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 18:00:44.142259 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:44.142230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 18:00:56.098850 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.098812    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 18:00:56.099416 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.099116    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" containerID="cri-o://497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac" gracePeriod=30
Jul 09 18:00:56.099416 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.099154    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kube-rbac-proxy" containerID="cri-o://2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf" gracePeriod=30
Jul 09 18:00:56.164871 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.164845    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:00:56.165208 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165196    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165244    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165249    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165258    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165265 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165263    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165535 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165310    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165535 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165321    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d67e88-66fd-44f1-96fa-0849e42332ac" containerName="kserve-container"
Jul 09 18:00:56.165535 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165327    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kube-rbac-proxy"
Jul 09 18:00:56.165535 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.165332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f243305b-4440-4c3d-bf34-ae1bcb2c2ec5" containerName="kserve-container"
Jul 09 18:00:56.169973 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.169957    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.173194 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.173177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53a2e-predictor-serving-cert\""
Jul 09 18:00:56.173194 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.173184    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53a2e-kube-rbac-proxy-sar-config\""
Jul 09 18:00:56.180935 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.180916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:00:56.191274 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.191251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpmd\" (UniqueName: \"kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.191403 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.191290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.191475 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.191419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.244606 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.244576    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 18:00:56.244807 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.244786    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" containerID="cri-o://be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b" gracePeriod=30
Jul 09 18:00:56.244884 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.244819    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kube-rbac-proxy" containerID="cri-o://bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf" gracePeriod=30
Jul 09 18:00:56.282661 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.282634    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:00:56.286271 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.286249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.289606 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.289588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53a2e-predictor-serving-cert\""
Jul 09 18:00:56.289606 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.289600    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53a2e-kube-rbac-proxy-sar-config\""
Jul 09 18:00:56.292260 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.292235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.292381 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.292322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpmd\" (UniqueName: \"kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.292446 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.292394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.292446 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:56.292416    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-serving-cert: secret "success-200-isvc-53a2e-predictor-serving-cert" not found
Jul 09 18:00:56.292536 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:56.292480    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls podName:69678cc8-3f8d-4d11-a05c-fcc8866d34f5 nodeName:}" failed. No retries permitted until 2026-07-09 18:00:56.7924585 +0000 UTC m=+1270.395343950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls") pod "success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" (UID: "69678cc8-3f8d-4d11-a05c-fcc8866d34f5") : secret "success-200-isvc-53a2e-predictor-serving-cert" not found
Jul 09 18:00:56.293152 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.293128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.298792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.298768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:00:56.309450 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.309427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpmd\" (UniqueName: \"kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.393440 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.393412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.393595 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.393473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.393595 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.393545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j764r\" (UniqueName: \"kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.398064 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.398041    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerID="2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf" exitCode=2
Jul 09 18:00:56.398182 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.398109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerDied","Data":"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"}
Jul 09 18:00:56.399818 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.399799    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerID="bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf" exitCode=2
Jul 09 18:00:56.399933 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.399866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerDied","Data":"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"}
Jul 09 18:00:56.494900 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.494871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.495006 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.494921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.495006 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.494950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j764r\" (UniqueName: \"kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.495618 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.495598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.497258 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.497236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.504840 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.504818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j764r\" (UniqueName: \"kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r\") pod \"error-404-isvc-53a2e-predictor-5ccf567585-c458v\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") " pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.597868 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.597836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:56.754802 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.754781    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:00:56.756892 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:00:56.756866    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad27de7_ff28_4b41_bce2_2552715cd5b4.slice/crio-aaa770597b63555c8b20c8d4c8866ccb1ea4b711e0b38033e3001b9ce18cf4c7 WatchSource:0}: Error finding container aaa770597b63555c8b20c8d4c8866ccb1ea4b711e0b38033e3001b9ce18cf4c7: Status 404 returned error can't find the container with id aaa770597b63555c8b20c8d4c8866ccb1ea4b711e0b38033e3001b9ce18cf4c7
Jul 09 18:00:56.797433 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.797407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:56.799941 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:56.799922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") pod \"success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") " pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:57.080734 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.080701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:57.220187 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.220162    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:00:57.222394 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:00:57.222349    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69678cc8_3f8d_4d11_a05c_fcc8866d34f5.slice/crio-c3c922e6a8a7a0eedd137aa0e51074a90752f2e42c7b9de0671e5de0d5eeaa73 WatchSource:0}: Error finding container c3c922e6a8a7a0eedd137aa0e51074a90752f2e42c7b9de0671e5de0d5eeaa73: Status 404 returned error can't find the container with id c3c922e6a8a7a0eedd137aa0e51074a90752f2e42c7b9de0671e5de0d5eeaa73
Jul 09 18:00:57.406063 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.406029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerStarted","Data":"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"}
Jul 09 18:00:57.406248 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.406093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerStarted","Data":"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"}
Jul 09 18:00:57.406248 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.406120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerStarted","Data":"c3c922e6a8a7a0eedd137aa0e51074a90752f2e42c7b9de0671e5de0d5eeaa73"}
Jul 09 18:00:57.406348 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.406265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:57.406348 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.406320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:00:57.407903 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.407879    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:00:57.408196 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.408173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerStarted","Data":"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"}
Jul 09 18:00:57.408285 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.408221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerStarted","Data":"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"}
Jul 09 18:00:57.408285 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.408242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerStarted","Data":"aaa770597b63555c8b20c8d4c8866ccb1ea4b711e0b38033e3001b9ce18cf4c7"}
Jul 09 18:00:57.408285 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.408265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:57.408423 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.408325    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:00:57.409440 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.409420    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:00:57.437067 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.437028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podStartSLOduration=1.437016053 podStartE2EDuration="1.437016053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:57.434324964 +0000 UTC m=+1271.037210421" watchObservedRunningTime="2026-07-09 18:00:57.437016053 +0000 UTC m=+1271.039901508"
Jul 09 18:00:57.456468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.456430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podStartSLOduration=1.456421121 podStartE2EDuration="1.456421121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:57.454796456 +0000 UTC m=+1271.057681913" watchObservedRunningTime="2026-07-09 18:00:57.456421121 +0000 UTC m=+1271.059306577"
Jul 09 18:00:57.914562 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.914520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 09 18:00:57.919942 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:57.919916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 09 18:00:58.413963 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:58.413922    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:00:58.414431 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:58.414052    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:00:58.919801 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:58.919753    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 09 18:00:58.924816 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:58.924785    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 09 18:00:59.342345 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.342323    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 18:00:59.416195 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.416171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w7xm\" (UniqueName: \"kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm\") pod \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") "
Jul 09 18:00:59.416589 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.416258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config\") pod \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") "
Jul 09 18:00:59.416589 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.416311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls\") pod \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\" (UID: \"9a229921-0ad5-4fc3-947f-4b0c2f8f029d\") "
Jul 09 18:00:59.416701 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.416628    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a229921-0ad5-4fc3-947f-4b0c2f8f029d" (UID: "9a229921-0ad5-4fc3-947f-4b0c2f8f029d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:00:59.418286 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.418262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a229921-0ad5-4fc3-947f-4b0c2f8f029d" (UID: "9a229921-0ad5-4fc3-947f-4b0c2f8f029d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:00:59.418286 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.418273    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm" (OuterVolumeSpecName: "kube-api-access-4w7xm") pod "9a229921-0ad5-4fc3-947f-4b0c2f8f029d" (UID: "9a229921-0ad5-4fc3-947f-4b0c2f8f029d"). InnerVolumeSpecName "kube-api-access-4w7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:00:59.420697 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.420673    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerID="497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac" exitCode=0
Jul 09 18:00:59.420773 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.420707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerDied","Data":"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"}
Jul 09 18:00:59.420773 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.420720    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"
Jul 09 18:00:59.420773 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.420759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn" event={"ID":"9a229921-0ad5-4fc3-947f-4b0c2f8f029d","Type":"ContainerDied","Data":"74bcbf435f0516d0e69dd2acfac286d078c66b1f07bfa41789b1b5ed1de5aad1"}
Jul 09 18:00:59.420873 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.420786    2577 scope.go:117] "RemoveContainer" containerID="2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"
Jul 09 18:00:59.457871 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.457852    2577 scope.go:117] "RemoveContainer" containerID="497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"
Jul 09 18:00:59.465404 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.465383    2577 scope.go:117] "RemoveContainer" containerID="2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"
Jul 09 18:00:59.465709 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:59.465692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf\": container with ID starting with 2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf not found: ID does not exist" containerID="2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"
Jul 09 18:00:59.465775 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.465718    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf"} err="failed to get container status \"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf\": rpc error: code = NotFound desc = could not find container \"2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf\": container with ID starting with 2d7530f03465443176d71ae5792a1410390c282804a8b44c8ffeedc53c24bfbf not found: ID does not exist"
Jul 09 18:00:59.465775 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.465737    2577 scope.go:117] "RemoveContainer" containerID="497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"
Jul 09 18:00:59.465999 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:00:59.465975    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac\": container with ID starting with 497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac not found: ID does not exist" containerID="497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"
Jul 09 18:00:59.466064 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.466010    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac"} err="failed to get container status \"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac\": rpc error: code = NotFound desc = could not find container \"497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac\": container with ID starting with 497db6229f17812de2d56bd21c3e97306fc7d43d833f63d7988d5bb2f3f401ac not found: ID does not exist"
Jul 09 18:00:59.470396 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.470375    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 18:00:59.476288 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.476264    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-71ac7-predictor-6899d48d94-czfzn"]
Jul 09 18:00:59.517896 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.517871    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:59.517993 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.517896    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w7xm\" (UniqueName: \"kubernetes.io/projected/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-api-access-4w7xm\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:59.517993 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.517913    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a229921-0ad5-4fc3-947f-4b0c2f8f029d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:59.592426 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.592397    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 18:00:59.618760 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.618735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swr69\" (UniqueName: \"kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69\") pod \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") "
Jul 09 18:00:59.618881 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.618833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config\") pod \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") "
Jul 09 18:00:59.618881 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.618862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls\") pod \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\" (UID: \"5c22b1bd-0f20-4bd8-8323-1467b767b87d\") "
Jul 09 18:00:59.619219 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.619193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c22b1bd-0f20-4bd8-8323-1467b767b87d" (UID: "5c22b1bd-0f20-4bd8-8323-1467b767b87d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:00:59.620739 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.620715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69" (OuterVolumeSpecName: "kube-api-access-swr69") pod "5c22b1bd-0f20-4bd8-8323-1467b767b87d" (UID: "5c22b1bd-0f20-4bd8-8323-1467b767b87d"). InnerVolumeSpecName "kube-api-access-swr69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:00:59.620881 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.620857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c22b1bd-0f20-4bd8-8323-1467b767b87d" (UID: "5c22b1bd-0f20-4bd8-8323-1467b767b87d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:00:59.720509 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.720436    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c22b1bd-0f20-4bd8-8323-1467b767b87d-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:59.720509 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.720462    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swr69\" (UniqueName: \"kubernetes.io/projected/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-api-access-swr69\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:00:59.720509 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:00:59.720472    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c22b1bd-0f20-4bd8-8323-1467b767b87d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:00.427971 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.427929    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerID="be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b" exitCode=0
Jul 09 18:01:00.428375 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.427982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerDied","Data":"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"}
Jul 09 18:01:00.428375 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.428064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb" event={"ID":"5c22b1bd-0f20-4bd8-8323-1467b767b87d","Type":"ContainerDied","Data":"4240e682553201cf03f4e4c0639a585fed3789cc07d80d8f88cafb5a64034129"}
Jul 09 18:01:00.428375 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.427992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"
Jul 09 18:01:00.428375 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.428093    2577 scope.go:117] "RemoveContainer" containerID="bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"
Jul 09 18:01:00.436841 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.436826    2577 scope.go:117] "RemoveContainer" containerID="be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"
Jul 09 18:01:00.446684 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.446665    2577 scope.go:117] "RemoveContainer" containerID="bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"
Jul 09 18:01:00.446934 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:00.446915    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf\": container with ID starting with bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf not found: ID does not exist" containerID="bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"
Jul 09 18:01:00.446986 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.446943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf"} err="failed to get container status \"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf\": rpc error: code = NotFound desc = could not find container \"bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf\": container with ID starting with bd26d2d8c45048381286271cf9f5b7bc2e3f6ff3eb7895d164205bc644ccaccf not found: ID does not exist"
Jul 09 18:01:00.446986 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.446960    2577 scope.go:117] "RemoveContainer" containerID="be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"
Jul 09 18:01:00.447155 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:00.447136    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b\": container with ID starting with be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b not found: ID does not exist" containerID="be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"
Jul 09 18:01:00.447192 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.447162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b"} err="failed to get container status \"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b\": rpc error: code = NotFound desc = could not find container \"be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b\": container with ID starting with be860f253bef35310fa58ede87cc65e05e7671680c47850db8a10ea85323cd1b not found: ID does not exist"
Jul 09 18:01:00.452493 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.452437    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 18:01:00.454249 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.454231    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-71ac7-predictor-866d48d8f9-r2xqb"]
Jul 09 18:01:00.968499 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.968471    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" path="/var/lib/kubelet/pods/5c22b1bd-0f20-4bd8-8323-1467b767b87d/volumes"
Jul 09 18:01:00.968943 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:00.968928    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" path="/var/lib/kubelet/pods/9a229921-0ad5-4fc3-947f-4b0c2f8f029d/volumes"
Jul 09 18:01:03.418540 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:03.418508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:01:03.419104 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:03.418813    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:01:03.419104 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:03.418963    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:01:03.419312 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:03.419283    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:01:06.290148 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.290112    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 18:01:06.290660 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.290430    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container" containerID="cri-o://49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb" gracePeriod=30
Jul 09 18:01:06.290660 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.290486    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kube-rbac-proxy" containerID="cri-o://2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b" gracePeriod=30
Jul 09 18:01:06.343213 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:01:06.343565 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343567    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343580    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343586    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343597    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343602    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container"
Jul 09 18:01:06.343617 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343614    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container"
Jul 09 18:01:06.343828 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343620    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container"
Jul 09 18:01:06.343828 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kserve-container"
Jul 09 18:01:06.343828 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343828 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343687    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a229921-0ad5-4fc3-947f-4b0c2f8f029d" containerName="kube-rbac-proxy"
Jul 09 18:01:06.343828 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.343693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c22b1bd-0f20-4bd8-8323-1467b767b87d" containerName="kserve-container"
Jul 09 18:01:06.348553 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.348534    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.351606 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.351587    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-14ef6-predictor-serving-cert\""
Jul 09 18:01:06.351713 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.351588    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-14ef6-kube-rbac-proxy-sar-config\""
Jul 09 18:01:06.357287 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.357266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:01:06.393939 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.393904    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 18:01:06.394226 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.394192    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container" containerID="cri-o://948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd" gracePeriod=30
Jul 09 18:01:06.394389 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.394267    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kube-rbac-proxy" containerID="cri-o://1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd" gracePeriod=30
Jul 09 18:01:06.457681 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.457641    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerID="2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b" exitCode=2
Jul 09 18:01:06.457841 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.457706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerDied","Data":"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"}
Jul 09 18:01:06.480267 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.480233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.480449 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.480299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsv6\" (UniqueName: \"kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.480449 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.480336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.506704 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.506675    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:01:06.510099 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.510080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.513166 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.513145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-14ef6-predictor-serving-cert\""
Jul 09 18:01:06.513269 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.513147    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-14ef6-kube-rbac-proxy-sar-config\""
Jul 09 18:01:06.523074 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.523054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:01:06.581626 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.581764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.581764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmm5h\" (UniqueName: \"kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.581764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsv6\" (UniqueName: \"kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.581764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.581927 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.581763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.581927 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:06.581887    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-serving-cert: secret "success-200-isvc-14ef6-predictor-serving-cert" not found
Jul 09 18:01:06.581997 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:06.581965    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls podName:ef22e18a-ec83-476f-b7ed-d4577bb5c748 nodeName:}" failed. No retries permitted until 2026-07-09 18:01:07.08194423 +0000 UTC m=+1280.684829678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls") pod "success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" (UID: "ef22e18a-ec83-476f-b7ed-d4577bb5c748") : secret "success-200-isvc-14ef6-predictor-serving-cert" not found
Jul 09 18:01:06.582296 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.582279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.593944 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.593912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsv6\" (UniqueName: \"kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:06.682314 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.682281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.682494 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.682394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmm5h\" (UniqueName: \"kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.682494 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.682466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.683053 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.683031    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.684780 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.684752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.691505 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.691471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmm5h\" (UniqueName: \"kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h\") pod \"error-404-isvc-14ef6-predictor-6478f88585-pt8tz\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") " pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.820684 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.820651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:06.956096 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:06.956075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:01:06.957575 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:01:06.957545    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1326a459_a070_4137_9574_a50ba77b6ef9.slice/crio-ba386c794bd5b49a47011966f73f205d36ced49891bdc4acdbfce3531966a788 WatchSource:0}: Error finding container ba386c794bd5b49a47011966f73f205d36ced49891bdc4acdbfce3531966a788: Status 404 returned error can't find the container with id ba386c794bd5b49a47011966f73f205d36ced49891bdc4acdbfce3531966a788
Jul 09 18:01:07.086295 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.086225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:07.089245 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.089213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") pod \"success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") " pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:07.260058 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.260030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:07.395941 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.395918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:01:07.397201 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:01:07.397174    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef22e18a_ec83_476f_b7ed_d4577bb5c748.slice/crio-6851a76ad9bac44a9453e94640f493afebaaf7ca26a4944a7d070dcaf1883c39 WatchSource:0}: Error finding container 6851a76ad9bac44a9453e94640f493afebaaf7ca26a4944a7d070dcaf1883c39: Status 404 returned error can't find the container with id 6851a76ad9bac44a9453e94640f493afebaaf7ca26a4944a7d070dcaf1883c39
Jul 09 18:01:07.465173 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.465143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerStarted","Data":"c970d26015e6156366d820e02b29b852bec61190371af9ffbdb2fa112a3643aa"}
Jul 09 18:01:07.465287 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.465211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerStarted","Data":"e62c2185a1c0850d4b281352d36aab851c87084f59fe8c913f9a0560863efc7d"}
Jul 09 18:01:07.465287 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.465237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerStarted","Data":"ba386c794bd5b49a47011966f73f205d36ced49891bdc4acdbfce3531966a788"}
Jul 09 18:01:07.465287 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.465268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:07.465507 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.465302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:07.466845 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.466818    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:07.467090 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.467072    2577 generic.go:356] "Generic (PLEG): container finished" podID="4277b038-4f02-471b-8133-e36bf4336e49" containerID="1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd" exitCode=2
Jul 09 18:01:07.467155 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.467140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerDied","Data":"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"}
Jul 09 18:01:07.468534 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.468514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerStarted","Data":"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"}
Jul 09 18:01:07.468621 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:07.468560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerStarted","Data":"6851a76ad9bac44a9453e94640f493afebaaf7ca26a4944a7d070dcaf1883c39"}
Jul 09 18:01:08.131215 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.131175    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 09 18:01:08.474986 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.474954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerStarted","Data":"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"}
Jul 09 18:01:08.475464 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.475304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:08.475464 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.475331    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:08.475464 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.475355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:08.476328 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.476306    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:08.495734 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.495691    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podStartSLOduration=2.495675339 podStartE2EDuration="2.495675339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:01:08.49464358 +0000 UTC m=+1282.097529066" watchObservedRunningTime="2026-07-09 18:01:08.495675339 +0000 UTC m=+1282.098560794"
Jul 09 18:01:08.495979 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:08.495955    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podStartSLOduration=2.495946297 podStartE2EDuration="2.495946297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:01:07.489888453 +0000 UTC m=+1281.092773924" watchObservedRunningTime="2026-07-09 18:01:08.495946297 +0000 UTC m=+1282.098831754"
Jul 09 18:01:09.136101 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:09.136056    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 09 18:01:09.480049 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:09.480010    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:10.061008 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.060981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 18:01:10.116039 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.116012    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config\") pod \"0f6f781e-b38d-4923-90dc-b95195ad9adb\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") "
Jul 09 18:01:10.116203 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.116059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls\") pod \"0f6f781e-b38d-4923-90dc-b95195ad9adb\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") "
Jul 09 18:01:10.116203 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.116103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m54j4\" (UniqueName: \"kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4\") pod \"0f6f781e-b38d-4923-90dc-b95195ad9adb\" (UID: \"0f6f781e-b38d-4923-90dc-b95195ad9adb\") "
Jul 09 18:01:10.116394 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.116354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f6f781e-b38d-4923-90dc-b95195ad9adb" (UID: "0f6f781e-b38d-4923-90dc-b95195ad9adb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:10.119414 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.119389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f6f781e-b38d-4923-90dc-b95195ad9adb" (UID: "0f6f781e-b38d-4923-90dc-b95195ad9adb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:10.119526 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.119452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4" (OuterVolumeSpecName: "kube-api-access-m54j4") pod "0f6f781e-b38d-4923-90dc-b95195ad9adb" (UID: "0f6f781e-b38d-4923-90dc-b95195ad9adb"). InnerVolumeSpecName "kube-api-access-m54j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:10.138878 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.138827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 18:01:10.217406 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config\") pod \"4277b038-4f02-471b-8133-e36bf4336e49\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") "
Jul 09 18:01:10.217555 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") pod \"4277b038-4f02-471b-8133-e36bf4336e49\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") "
Jul 09 18:01:10.217555 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfwr7\" (UniqueName: \"kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7\") pod \"4277b038-4f02-471b-8133-e36bf4336e49\" (UID: \"4277b038-4f02-471b-8133-e36bf4336e49\") "
Jul 09 18:01:10.217683 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217661    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4277b038-4f02-471b-8133-e36bf4336e49" (UID: "4277b038-4f02-471b-8133-e36bf4336e49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:01:10.217785 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217772    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.217817 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217791    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f6f781e-b38d-4923-90dc-b95195ad9adb-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.217817 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217800    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54j4\" (UniqueName: \"kubernetes.io/projected/0f6f781e-b38d-4923-90dc-b95195ad9adb-kube-api-access-m54j4\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.217817 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.217808    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4277b038-4f02-471b-8133-e36bf4336e49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.219508 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.219488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4277b038-4f02-471b-8133-e36bf4336e49" (UID: "4277b038-4f02-471b-8133-e36bf4336e49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:10.219559 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.219506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7" (OuterVolumeSpecName: "kube-api-access-lfwr7") pod "4277b038-4f02-471b-8133-e36bf4336e49" (UID: "4277b038-4f02-471b-8133-e36bf4336e49"). InnerVolumeSpecName "kube-api-access-lfwr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:10.318401 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.318347    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwr7\" (UniqueName: \"kubernetes.io/projected/4277b038-4f02-471b-8133-e36bf4336e49-kube-api-access-lfwr7\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.318401 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.318399    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4277b038-4f02-471b-8133-e36bf4336e49-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:01:10.486690 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.486641    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerID="49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb" exitCode=0
Jul 09 18:01:10.487172 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.486709    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"
Jul 09 18:01:10.487172 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.486810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerDied","Data":"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"}
Jul 09 18:01:10.487172 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.486873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz" event={"ID":"0f6f781e-b38d-4923-90dc-b95195ad9adb","Type":"ContainerDied","Data":"f6b641336d38dd60ab10a31426043b2af199641a2af48eeb0a68b05c7a93cf88"}
Jul 09 18:01:10.487172 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.486918    2577 scope.go:117] "RemoveContainer" containerID="2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"
Jul 09 18:01:10.489783 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.489747    2577 generic.go:356] "Generic (PLEG): container finished" podID="4277b038-4f02-471b-8133-e36bf4336e49" containerID="948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd" exitCode=0
Jul 09 18:01:10.489884 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.489820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerDied","Data":"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"}
Jul 09 18:01:10.489941 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.489893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn" event={"ID":"4277b038-4f02-471b-8133-e36bf4336e49","Type":"ContainerDied","Data":"87fdc7fa0b4773a13e8eda9f039cb280b3a966b44c21b44357296d3fe8acade7"}
Jul 09 18:01:10.489991 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.489956    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"
Jul 09 18:01:10.499175 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.499156    2577 scope.go:117] "RemoveContainer" containerID="49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"
Jul 09 18:01:10.506712 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.506694    2577 scope.go:117] "RemoveContainer" containerID="2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"
Jul 09 18:01:10.506948 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:10.506930    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b\": container with ID starting with 2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b not found: ID does not exist" containerID="2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"
Jul 09 18:01:10.507008 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.506958    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b"} err="failed to get container status \"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b\": rpc error: code = NotFound desc = could not find container \"2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b\": container with ID starting with 2eb475134e0efee8856c72c9ddadbe0efce93fb3899e5f15501e026bb6c27f6b not found: ID does not exist"
Jul 09 18:01:10.507008 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.506975    2577 scope.go:117] "RemoveContainer" containerID="49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"
Jul 09 18:01:10.507218 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:10.507200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb\": container with ID starting with 49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb not found: ID does not exist" containerID="49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"
Jul 09 18:01:10.507263 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.507223    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb"} err="failed to get container status \"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb\": rpc error: code = NotFound desc = could not find container \"49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb\": container with ID starting with 49ec2f819d126ded615847596694b492fee66e2cd4995d849a1dbd84623ca7bb not found: ID does not exist"
Jul 09 18:01:10.507263 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.507237    2577 scope.go:117] "RemoveContainer" containerID="1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"
Jul 09 18:01:10.513379 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.513341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 18:01:10.515061 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.515037    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5bd2e-predictor-66fc6dbb48-nndpz"]
Jul 09 18:01:10.515585 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.515570    2577 scope.go:117] "RemoveContainer" containerID="948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"
Jul 09 18:01:10.523114 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.523100    2577 scope.go:117] "RemoveContainer" containerID="1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"
Jul 09 18:01:10.523341 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:10.523324    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd\": container with ID starting with 1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd not found: ID does not exist" containerID="1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"
Jul 09 18:01:10.523398 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.523347    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd"} err="failed to get container status \"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd\": rpc error: code = NotFound desc = could not find container \"1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd\": container with ID starting with 1d0965d05587f7894bbde04ae0841f2f13c7f909652db2468c500fae69207ffd not found: ID does not exist"
Jul 09 18:01:10.523398 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.523374    2577 scope.go:117] "RemoveContainer" containerID="948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"
Jul 09 18:01:10.523627 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:01:10.523614    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd\": container with ID starting with 948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd not found: ID does not exist" containerID="948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"
Jul 09 18:01:10.523675 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.523632    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd"} err="failed to get container status \"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd\": rpc error: code = NotFound desc = could not find container \"948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd\": container with ID starting with 948b334fd453f848a4fc14a826384ef458b6a2fa837a16efb2e74e642a37afcd not found: ID does not exist"
Jul 09 18:01:10.526389 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.526352    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 18:01:10.528444 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.528425    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5bd2e-predictor-659bf7cb45-kwlgn"]
Jul 09 18:01:10.967232 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.967193    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" path="/var/lib/kubelet/pods/0f6f781e-b38d-4923-90dc-b95195ad9adb/volumes"
Jul 09 18:01:10.967671 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:10.967656    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4277b038-4f02-471b-8133-e36bf4336e49" path="/var/lib/kubelet/pods/4277b038-4f02-471b-8133-e36bf4336e49/volumes"
Jul 09 18:01:13.419688 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:13.419655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:01:13.420083 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:13.419653    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:01:13.479379 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:13.479341    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:13.479933 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:13.479909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:14.483957 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:14.483929    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:01:14.484602 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:14.484566    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:23.419922 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:23.419876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:01:23.420316 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:23.419876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:01:23.480749 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:23.480719    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:24.484826 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:24.484780    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:33.419807 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:33.419762    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 09 18:01:33.420236 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:33.419762    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 09 18:01:33.480577 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:33.480539    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:34.484695 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:34.484643    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:43.420128 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:43.420096    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:01:43.420629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:43.420608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:01:43.480768 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:43.480730    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:01:44.484995 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:44.484951    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:01:53.480944 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:53.480911    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:01:54.485590 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:01:54.485553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:02:06.388036 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.387996    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:02:06.388509 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.388278    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container" containerID="cri-o://41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6" gracePeriod=30
Jul 09 18:02:06.388509 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.388482    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kube-rbac-proxy" containerID="cri-o://92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891" gracePeriod=30
Jul 09 18:02:06.549044 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.549012    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:02:06.549279 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.549235    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container" containerID="cri-o://74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502" gracePeriod=30
Jul 09 18:02:06.549279 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.549265    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kube-rbac-proxy" containerID="cri-o://c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a" gracePeriod=30
Jul 09 18:02:06.655995 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.655912    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:02:06.656592 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656569    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container"
Jul 09 18:02:06.656592 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656593    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656607    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kube-rbac-proxy"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656616    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kube-rbac-proxy"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656642    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kube-rbac-proxy"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656652    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kube-rbac-proxy"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656679    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656688    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kube-rbac-proxy"
Jul 09 18:02:06.656792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656793    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kserve-container"
Jul 09 18:02:06.657163 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656804    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6f781e-b38d-4923-90dc-b95195ad9adb" containerName="kserve-container"
Jul 09 18:02:06.657163 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.656813    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4277b038-4f02-471b-8133-e36bf4336e49" containerName="kube-rbac-proxy"
Jul 09 18:02:06.660114 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.660092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.663351 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.663333    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67922-predictor-serving-cert\""
Jul 09 18:02:06.663448 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.663388    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67922-kube-rbac-proxy-sar-config\""
Jul 09 18:02:06.698356 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.698329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:02:06.750152 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.750119    2577 generic.go:356] "Generic (PLEG): container finished" podID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerID="92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891" exitCode=2
Jul 09 18:02:06.750309 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.750195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerDied","Data":"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"}
Jul 09 18:02:06.752238 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.752211    2577 generic.go:356] "Generic (PLEG): container finished" podID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerID="c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a" exitCode=2
Jul 09 18:02:06.752353 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.752275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerDied","Data":"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"}
Jul 09 18:02:06.792250 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.792215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.792438 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.792276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.792495 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.792424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlsbx\" (UniqueName: \"kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.807889 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.807855    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:02:06.811440 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.811423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.814290 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.814266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67922-kube-rbac-proxy-sar-config\""
Jul 09 18:02:06.814425 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.814266    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67922-predictor-serving-cert\""
Jul 09 18:02:06.824334 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.824313    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:02:06.893489 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpnd\" (UniqueName: \"kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.893702 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.893702 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.893702 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.893702 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.893702 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.893699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlsbx\" (UniqueName: \"kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.894317 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.894288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.896109 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.896088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.905894 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.905865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlsbx\" (UniqueName: \"kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx\") pod \"success-200-isvc-67922-predictor-5948d8dd68-jknmx\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") " pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.969612 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.969580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:06.995054 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.995024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.995222 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.995085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpnd\" (UniqueName: \"kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.995222 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.995133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:06.995333 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:06.995231    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-67922-predictor-serving-cert: secret "error-404-isvc-67922-predictor-serving-cert" not found
Jul 09 18:02:06.995333 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:06.995323    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls podName:6a8237ad-4fb6-443d-a9a7-54f8141e16b3 nodeName:}" failed. No retries permitted until 2026-07-09 18:02:07.495300375 +0000 UTC m=+1341.098185829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls") pod "error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" (UID: "6a8237ad-4fb6-443d-a9a7-54f8141e16b3") : secret "error-404-isvc-67922-predictor-serving-cert" not found
Jul 09 18:02:06.995808 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:06.995789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:07.005138 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.005107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpnd\" (UniqueName: \"kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:07.110835 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.110805    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:02:07.112465 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:02:07.112437    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bad1b4_d188_4ad0_9ef0_6b8ab06193d4.slice/crio-b614710adb4297bebb6066b5a036e5a5cf03b24157b4bea20b792e4e7fd95755 WatchSource:0}: Error finding container b614710adb4297bebb6066b5a036e5a5cf03b24157b4bea20b792e4e7fd95755: Status 404 returned error can't find the container with id b614710adb4297bebb6066b5a036e5a5cf03b24157b4bea20b792e4e7fd95755
Jul 09 18:02:07.500031 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.499944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:07.502293 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.502269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") pod \"error-404-isvc-67922-predictor-5b8d4977c7-k7vt7\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") " pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:07.721445 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.721398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:07.758682 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.758589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerStarted","Data":"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"}
Jul 09 18:02:07.758682 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.758658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerStarted","Data":"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"}
Jul 09 18:02:07.758872 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.758687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerStarted","Data":"b614710adb4297bebb6066b5a036e5a5cf03b24157b4bea20b792e4e7fd95755"}
Jul 09 18:02:07.758872 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.758720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:07.758872 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.758750    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:07.760927 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.760585    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:07.781799 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.781745    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podStartSLOduration=1.781726793 podStartE2EDuration="1.781726793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:02:07.780685448 +0000 UTC m=+1341.383570905" watchObservedRunningTime="2026-07-09 18:02:07.781726793 +0000 UTC m=+1341.384612249"
Jul 09 18:02:07.863659 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:07.863632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:02:07.865267 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:02:07.865235    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8237ad_4fb6_443d_a9a7_54f8141e16b3.slice/crio-1a7715ef898b7a56ed77f53b2c88e9a312d56a926bbd152d1eec2a103b4ac46d WatchSource:0}: Error finding container 1a7715ef898b7a56ed77f53b2c88e9a312d56a926bbd152d1eec2a103b4ac46d: Status 404 returned error can't find the container with id 1a7715ef898b7a56ed77f53b2c88e9a312d56a926bbd152d1eec2a103b4ac46d
Jul 09 18:02:08.414699 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.414658    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 09 18:02:08.414876 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.414664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 09 18:02:08.765076 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.765044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerStarted","Data":"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"}
Jul 09 18:02:08.765482 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.765104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerStarted","Data":"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"}
Jul 09 18:02:08.765482 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.765133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerStarted","Data":"1a7715ef898b7a56ed77f53b2c88e9a312d56a926bbd152d1eec2a103b4ac46d"}
Jul 09 18:02:08.765482 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.765356    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:08.765482 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.765391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:08.790050 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:08.789976    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podStartSLOduration=2.789949541 podStartE2EDuration="2.789949541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:02:08.788409118 +0000 UTC m=+1342.391294575" watchObservedRunningTime="2026-07-09 18:02:08.789949541 +0000 UTC m=+1342.392835002"
Jul 09 18:02:09.750412 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.750387    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:02:09.772748 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.772718    2577 generic.go:356] "Generic (PLEG): container finished" podID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerID="41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6" exitCode=0
Jul 09 18:02:09.773189 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.772765    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"
Jul 09 18:02:09.773189 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.772787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerDied","Data":"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"}
Jul 09 18:02:09.773189 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.772853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv" event={"ID":"69678cc8-3f8d-4d11-a05c-fcc8866d34f5","Type":"ContainerDied","Data":"c3c922e6a8a7a0eedd137aa0e51074a90752f2e42c7b9de0671e5de0d5eeaa73"}
Jul 09 18:02:09.773189 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.772893    2577 scope.go:117] "RemoveContainer" containerID="92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"
Jul 09 18:02:09.773189 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.773103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:09.774816 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.774786    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:09.781586 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.781569    2577 scope.go:117] "RemoveContainer" containerID="41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"
Jul 09 18:02:09.789677 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.789650    2577 scope.go:117] "RemoveContainer" containerID="92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"
Jul 09 18:02:09.789992 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:09.789970    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891\": container with ID starting with 92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891 not found: ID does not exist" containerID="92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"
Jul 09 18:02:09.790078 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.790010    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891"} err="failed to get container status \"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891\": rpc error: code = NotFound desc = could not find container \"92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891\": container with ID starting with 92dcacdbe471c0c26a746f8d34ed299bab161cc08b72384a883b8507c186e891 not found: ID does not exist"
Jul 09 18:02:09.790078 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.790034    2577 scope.go:117] "RemoveContainer" containerID="41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"
Jul 09 18:02:09.790320 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:09.790299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6\": container with ID starting with 41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6 not found: ID does not exist" containerID="41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"
Jul 09 18:02:09.790400 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.790328    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6"} err="failed to get container status \"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6\": rpc error: code = NotFound desc = could not find container \"41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6\": container with ID starting with 41e43b83484eb6390d0517cf6b1a2a90e14526c87d606d61232a64bd5f8af1f6 not found: ID does not exist"
Jul 09 18:02:09.820310 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.820286    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhpmd\" (UniqueName: \"kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd\") pod \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") "
Jul 09 18:02:09.820486 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.820346    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config\") pod \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") "
Jul 09 18:02:09.820486 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.820399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") pod \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\" (UID: \"69678cc8-3f8d-4d11-a05c-fcc8866d34f5\") "
Jul 09 18:02:09.820692 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.820666    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69678cc8-3f8d-4d11-a05c-fcc8866d34f5" (UID: "69678cc8-3f8d-4d11-a05c-fcc8866d34f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:02:09.884920 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.884828    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd" (OuterVolumeSpecName: "kube-api-access-mhpmd") pod "69678cc8-3f8d-4d11-a05c-fcc8866d34f5" (UID: "69678cc8-3f8d-4d11-a05c-fcc8866d34f5"). InnerVolumeSpecName "kube-api-access-mhpmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:02:09.884920 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.884840    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69678cc8-3f8d-4d11-a05c-fcc8866d34f5" (UID: "69678cc8-3f8d-4d11-a05c-fcc8866d34f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:02:09.921498 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.921472    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:09.921498 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.921496    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:09.921634 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:09.921505    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpmd\" (UniqueName: \"kubernetes.io/projected/69678cc8-3f8d-4d11-a05c-fcc8866d34f5-kube-api-access-mhpmd\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:10.097437 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.097400    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:02:10.100645 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.100619    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53a2e-predictor-57fccffd9b-6ljrv"]
Jul 09 18:02:10.107941 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.107918    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:02:10.227686 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.227654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config\") pod \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") "
Jul 09 18:02:10.227857 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.227711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls\") pod \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") "
Jul 09 18:02:10.227857 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.227771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j764r\" (UniqueName: \"kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r\") pod \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\" (UID: \"1ad27de7-ff28-4b41-bce2-2552715cd5b4\") "
Jul 09 18:02:10.228016 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.227984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ad27de7-ff28-4b41-bce2-2552715cd5b4" (UID: "1ad27de7-ff28-4b41-bce2-2552715cd5b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:02:10.229793 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.229773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r" (OuterVolumeSpecName: "kube-api-access-j764r") pod "1ad27de7-ff28-4b41-bce2-2552715cd5b4" (UID: "1ad27de7-ff28-4b41-bce2-2552715cd5b4"). InnerVolumeSpecName "kube-api-access-j764r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:02:10.229875 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.229790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ad27de7-ff28-4b41-bce2-2552715cd5b4" (UID: "1ad27de7-ff28-4b41-bce2-2552715cd5b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:02:10.329124 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.329093    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j764r\" (UniqueName: \"kubernetes.io/projected/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-api-access-j764r\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:10.329124 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.329119    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ad27de7-ff28-4b41-bce2-2552715cd5b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:10.329124 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.329129    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ad27de7-ff28-4b41-bce2-2552715cd5b4-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:02:10.779507 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.779472    2577 generic.go:356] "Generic (PLEG): container finished" podID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerID="74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502" exitCode=0
Jul 09 18:02:10.779907 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.779534    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"
Jul 09 18:02:10.779907 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.779552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerDied","Data":"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"}
Jul 09 18:02:10.779907 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.779637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v" event={"ID":"1ad27de7-ff28-4b41-bce2-2552715cd5b4","Type":"ContainerDied","Data":"aaa770597b63555c8b20c8d4c8866ccb1ea4b711e0b38033e3001b9ce18cf4c7"}
Jul 09 18:02:10.779907 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.779668    2577 scope.go:117] "RemoveContainer" containerID="c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"
Jul 09 18:02:10.780292 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.780258    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:10.789462 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.789443    2577 scope.go:117] "RemoveContainer" containerID="74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"
Jul 09 18:02:10.798048 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.798033    2577 scope.go:117] "RemoveContainer" containerID="c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"
Jul 09 18:02:10.798294 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:10.798276    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a\": container with ID starting with c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a not found: ID does not exist" containerID="c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"
Jul 09 18:02:10.798341 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.798301    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a"} err="failed to get container status \"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a\": rpc error: code = NotFound desc = could not find container \"c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a\": container with ID starting with c24081bfe40bfef9669bb87debb2b4e3d16e72103bcc2a39484aecbfcf6cf66a not found: ID does not exist"
Jul 09 18:02:10.798341 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.798316    2577 scope.go:117] "RemoveContainer" containerID="74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"
Jul 09 18:02:10.798563 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:02:10.798547    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502\": container with ID starting with 74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502 not found: ID does not exist" containerID="74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"
Jul 09 18:02:10.798616 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.798564    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502"} err="failed to get container status \"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502\": rpc error: code = NotFound desc = could not find container \"74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502\": container with ID starting with 74154ec5a018c66845c298264b6f0df130f1849507f3ed6d63ba825cbe89b502 not found: ID does not exist"
Jul 09 18:02:10.801607 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.801587    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:02:10.805286 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.805266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a2e-predictor-5ccf567585-c458v"]
Jul 09 18:02:10.966406 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.966380    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" path="/var/lib/kubelet/pods/1ad27de7-ff28-4b41-bce2-2552715cd5b4/volumes"
Jul 09 18:02:10.966822 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:10.966808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" path="/var/lib/kubelet/pods/69678cc8-3f8d-4d11-a05c-fcc8866d34f5/volumes"
Jul 09 18:02:13.771206 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:13.771178    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:13.771817 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:13.771791    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:15.784608 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:15.784577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:02:15.785191 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:15.785072    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:23.772423 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:23.772345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:25.785411 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:25.785353    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:33.772408 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:33.772340    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:35.785380 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:35.785319    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:43.772585 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:43.772536    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:02:45.785930 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:45.785890    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 09 18:02:53.772518 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:53.772491    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:02:55.786055 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:02:55.786027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:04:48.397499 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:04:48.397380    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:04:48.402505 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:04:48.402485    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:09:48.424829 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:09:48.424724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:09:48.431448 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:09:48.429488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:10:21.051626 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.051595    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:10:21.052128 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.051810    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" containerID="cri-o://88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b" gracePeriod=30
Jul 09 18:10:21.052128 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.051854    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kube-rbac-proxy" containerID="cri-o://8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e" gracePeriod=30
Jul 09 18:10:21.138978 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.138945    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:10:21.139195 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.139167    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" containerID="cri-o://e62c2185a1c0850d4b281352d36aab851c87084f59fe8c913f9a0560863efc7d" gracePeriod=30
Jul 09 18:10:21.139308 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.139211    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kube-rbac-proxy" containerID="cri-o://c970d26015e6156366d820e02b29b852bec61190371af9ffbdb2fa112a3643aa" gracePeriod=30
Jul 09 18:10:21.206856 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.206822    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:10:21.207248 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207234    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container"
Jul 09 18:10:21.207302 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207250    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container"
Jul 09 18:10:21.207302 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207280    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207302 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207286    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207302 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207319    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207324    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207417    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207428    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69678cc8-3f8d-4d11-a05c-fcc8866d34f5" containerName="kserve-container"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207436    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kube-rbac-proxy"
Jul 09 18:10:21.207468 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.207448    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad27de7-ff28-4b41-bce2-2552715cd5b4" containerName="kserve-container"
Jul 09 18:10:21.210292 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.210277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.214402 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.214378    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07956-kube-rbac-proxy-sar-config\""
Jul 09 18:10:21.214537 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.214405    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-07956-predictor-serving-cert\""
Jul 09 18:10:21.220773 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.220748    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:10:21.275954 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.275925    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:10:21.279395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.279378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.282223 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.282197    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-07956-kube-rbac-proxy-sar-config\""
Jul 09 18:10:21.282341 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.282250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-07956-predictor-serving-cert\""
Jul 09 18:10:21.292040 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.292014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:10:21.335209 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.335130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncpb\" (UniqueName: \"kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.335350 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.335251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.335439 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.335406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.435974 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.435929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.436176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.435995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.436176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.436031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.436176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.436054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5sm\" (UniqueName: \"kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.436176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.436071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.436176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.436123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncpb\" (UniqueName: \"kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.436613 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.436591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.438648 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.438624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.444402 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.444354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncpb\" (UniqueName: \"kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb\") pod \"success-200-isvc-07956-predictor-649f94cfd6-mgsrn\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") " pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.523606 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.523569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.536978 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.536954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5sm\" (UniqueName: \"kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.537092 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.537022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.537092 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.537065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.537809 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.537782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.539929 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.539894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.545471 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.545449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5sm\" (UniqueName: \"kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm\") pod \"error-404-isvc-07956-predictor-7d74d5c7dc-zctn2\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") " pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.591508 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.591434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.660796 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.660763    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:10:21.662725 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:10:21.662699    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf37bea_70f1_4e36_9e83_a39881f93292.slice/crio-6faf8e17419b99501cdc786230fb2e1b84195491d9f58523cfe377424e79d6db WatchSource:0}: Error finding container 6faf8e17419b99501cdc786230fb2e1b84195491d9f58523cfe377424e79d6db: Status 404 returned error can't find the container with id 6faf8e17419b99501cdc786230fb2e1b84195491d9f58523cfe377424e79d6db
Jul 09 18:10:21.743201 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.743177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:10:21.753515 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:10:21.753490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4857bdc5_980d_47e0_aff9_fdd41039776b.slice/crio-5683f259cfabdf7eeef31fdecb7369cb087918376db0d3b48f3684abfa293810 WatchSource:0}: Error finding container 5683f259cfabdf7eeef31fdecb7369cb087918376db0d3b48f3684abfa293810: Status 404 returned error can't find the container with id 5683f259cfabdf7eeef31fdecb7369cb087918376db0d3b48f3684abfa293810
Jul 09 18:10:21.979119 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.979084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerStarted","Data":"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"}
Jul 09 18:10:21.979307 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.979139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerStarted","Data":"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"}
Jul 09 18:10:21.979307 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.979170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerStarted","Data":"5683f259cfabdf7eeef31fdecb7369cb087918376db0d3b48f3684abfa293810"}
Jul 09 18:10:21.979307 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.979262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:21.980880 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.980855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerStarted","Data":"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"}
Jul 09 18:10:21.980989 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.980905    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.980989 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.980934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerStarted","Data":"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"}
Jul 09 18:10:21.980989 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.980965    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:21.980989 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.980985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerStarted","Data":"6faf8e17419b99501cdc786230fb2e1b84195491d9f58523cfe377424e79d6db"}
Jul 09 18:10:21.982432 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.982408    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:21.982757 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.982741    2577 generic.go:356] "Generic (PLEG): container finished" podID="1326a459-a070-4137-9574-a50ba77b6ef9" containerID="c970d26015e6156366d820e02b29b852bec61190371af9ffbdb2fa112a3643aa" exitCode=2
Jul 09 18:10:21.982825 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.982808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerDied","Data":"c970d26015e6156366d820e02b29b852bec61190371af9ffbdb2fa112a3643aa"}
Jul 09 18:10:21.984448 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.984391    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerID="8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e" exitCode=2
Jul 09 18:10:21.984554 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.984487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerDied","Data":"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"}
Jul 09 18:10:21.999135 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:21.999090    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podStartSLOduration=0.999075676 podStartE2EDuration="999.075676ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:21.998305644 +0000 UTC m=+1835.601191101" watchObservedRunningTime="2026-07-09 18:10:21.999075676 +0000 UTC m=+1835.601961131"
Jul 09 18:10:22.019157 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:22.019109    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podStartSLOduration=1.019093254 podStartE2EDuration="1.019093254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:22.018149082 +0000 UTC m=+1835.621034561" watchObservedRunningTime="2026-07-09 18:10:22.019093254 +0000 UTC m=+1835.621978710"
Jul 09 18:10:22.991143 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:22.990909    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:22.991143 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:22.991044    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:22.992062 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:22.992033    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:10:23.475402 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:23.475351    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 09 18:10:23.480738 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:23.480712    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 09 18:10:23.997444 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:23.997410    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:10:24.480559 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.480526    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 09 18:10:24.484903 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.484866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 09 18:10:24.660107 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.660050    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:10:24.762250 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.762221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") pod \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") "
Jul 09 18:10:24.762429 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.762270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config\") pod \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") "
Jul 09 18:10:24.762429 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.762315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgsv6\" (UniqueName: \"kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6\") pod \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\" (UID: \"ef22e18a-ec83-476f-b7ed-d4577bb5c748\") "
Jul 09 18:10:24.762671 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.762638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef22e18a-ec83-476f-b7ed-d4577bb5c748" (UID: "ef22e18a-ec83-476f-b7ed-d4577bb5c748"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:24.764301 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.764281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef22e18a-ec83-476f-b7ed-d4577bb5c748" (UID: "ef22e18a-ec83-476f-b7ed-d4577bb5c748"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:24.764386 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.764335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6" (OuterVolumeSpecName: "kube-api-access-xgsv6") pod "ef22e18a-ec83-476f-b7ed-d4577bb5c748" (UID: "ef22e18a-ec83-476f-b7ed-d4577bb5c748"). InnerVolumeSpecName "kube-api-access-xgsv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:24.863350 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.863320    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgsv6\" (UniqueName: \"kubernetes.io/projected/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-api-access-xgsv6\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:24.863350 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.863344    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef22e18a-ec83-476f-b7ed-d4577bb5c748-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:24.863350 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:24.863355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef22e18a-ec83-476f-b7ed-d4577bb5c748-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:25.011431 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.011393    2577 generic.go:356] "Generic (PLEG): container finished" podID="1326a459-a070-4137-9574-a50ba77b6ef9" containerID="e62c2185a1c0850d4b281352d36aab851c87084f59fe8c913f9a0560863efc7d" exitCode=0
Jul 09 18:10:25.011837 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.011455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerDied","Data":"e62c2185a1c0850d4b281352d36aab851c87084f59fe8c913f9a0560863efc7d"}
Jul 09 18:10:25.013077 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.013050    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerID="88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b" exitCode=0
Jul 09 18:10:25.013230 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.013105    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"
Jul 09 18:10:25.013230 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.013127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerDied","Data":"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"}
Jul 09 18:10:25.013230 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.013179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m" event={"ID":"ef22e18a-ec83-476f-b7ed-d4577bb5c748","Type":"ContainerDied","Data":"6851a76ad9bac44a9453e94640f493afebaaf7ca26a4944a7d070dcaf1883c39"}
Jul 09 18:10:25.013230 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.013205    2577 scope.go:117] "RemoveContainer" containerID="8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"
Jul 09 18:10:25.022543 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.022523    2577 scope.go:117] "RemoveContainer" containerID="88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"
Jul 09 18:10:25.031835 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.031808    2577 scope.go:117] "RemoveContainer" containerID="8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"
Jul 09 18:10:25.032394 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:10:25.032312    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e\": container with ID starting with 8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e not found: ID does not exist" containerID="8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"
Jul 09 18:10:25.032521 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.032389    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e"} err="failed to get container status \"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e\": rpc error: code = NotFound desc = could not find container \"8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e\": container with ID starting with 8a3bc34170dafb30a4219b09df6aebbff98e27e81fc3c9bf690cfa82b7d6407e not found: ID does not exist"
Jul 09 18:10:25.032521 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.032416    2577 scope.go:117] "RemoveContainer" containerID="88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"
Jul 09 18:10:25.032743 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:10:25.032721    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b\": container with ID starting with 88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b not found: ID does not exist" containerID="88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"
Jul 09 18:10:25.032833 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.032752    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b"} err="failed to get container status \"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b\": rpc error: code = NotFound desc = could not find container \"88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b\": container with ID starting with 88de5c257bcc66d75670f60bfcc7e3c3ed15a43e44c1dec8786c42afef81107b not found: ID does not exist"
Jul 09 18:10:25.034818 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.034793    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:10:25.039462 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.039403    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-14ef6-predictor-57b7ffcc88-b7n8m"]
Jul 09 18:10:25.101752 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.101726    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:10:25.165307 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.165278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmm5h\" (UniqueName: \"kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h\") pod \"1326a459-a070-4137-9574-a50ba77b6ef9\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") "
Jul 09 18:10:25.165307 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.165313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config\") pod \"1326a459-a070-4137-9574-a50ba77b6ef9\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") "
Jul 09 18:10:25.165533 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.165396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls\") pod \"1326a459-a070-4137-9574-a50ba77b6ef9\" (UID: \"1326a459-a070-4137-9574-a50ba77b6ef9\") "
Jul 09 18:10:25.165727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.165705    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1326a459-a070-4137-9574-a50ba77b6ef9" (UID: "1326a459-a070-4137-9574-a50ba77b6ef9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:10:25.167346 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.167327    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1326a459-a070-4137-9574-a50ba77b6ef9" (UID: "1326a459-a070-4137-9574-a50ba77b6ef9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:10:25.167426 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.167391    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h" (OuterVolumeSpecName: "kube-api-access-kmm5h") pod "1326a459-a070-4137-9574-a50ba77b6ef9" (UID: "1326a459-a070-4137-9574-a50ba77b6ef9"). InnerVolumeSpecName "kube-api-access-kmm5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:10:25.266430 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.266313    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmm5h\" (UniqueName: \"kubernetes.io/projected/1326a459-a070-4137-9574-a50ba77b6ef9-kube-api-access-kmm5h\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:25.266430 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.266352    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1326a459-a070-4137-9574-a50ba77b6ef9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:25.266430 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:25.266389    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1326a459-a070-4137-9574-a50ba77b6ef9-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:10:26.019517 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.019486    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"
Jul 09 18:10:26.019966 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.019487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz" event={"ID":"1326a459-a070-4137-9574-a50ba77b6ef9","Type":"ContainerDied","Data":"ba386c794bd5b49a47011966f73f205d36ced49891bdc4acdbfce3531966a788"}
Jul 09 18:10:26.019966 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.019634    2577 scope.go:117] "RemoveContainer" containerID="c970d26015e6156366d820e02b29b852bec61190371af9ffbdb2fa112a3643aa"
Jul 09 18:10:26.028122 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.028107    2577 scope.go:117] "RemoveContainer" containerID="e62c2185a1c0850d4b281352d36aab851c87084f59fe8c913f9a0560863efc7d"
Jul 09 18:10:26.044585 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.044558    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:10:26.046996 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.046972    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-14ef6-predictor-6478f88585-pt8tz"]
Jul 09 18:10:26.967002 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.966967    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" path="/var/lib/kubelet/pods/1326a459-a070-4137-9574-a50ba77b6ef9/volumes"
Jul 09 18:10:26.967449 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:26.967434    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" path="/var/lib/kubelet/pods/ef22e18a-ec83-476f-b7ed-d4577bb5c748/volumes"
Jul 09 18:10:27.995127 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:27.995100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:10:27.995729 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:27.995702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:29.001478 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:29.001454    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:10:29.002005 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:29.001979    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:10:37.995977 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:37.995939    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:39.002527 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:39.002483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:10:47.996223 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:47.996181    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:49.001987 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:49.001953    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:10:57.996429 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:57.996389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 09 18:10:59.002219 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:10:59.002178    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 09 18:11:07.996738 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:07.996708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:11:09.003534 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:09.003501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:11:21.263878 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.263847    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:11:21.264310 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.264074    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" containerID="cri-o://b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b" gracePeriod=30
Jul 09 18:11:21.264310 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.264140    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kube-rbac-proxy" containerID="cri-o://f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be" gracePeriod=30
Jul 09 18:11:21.320992 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.320959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:11:21.321212 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.321189    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container" containerID="cri-o://3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60" gracePeriod=30
Jul 09 18:11:21.321275 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.321218    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kube-rbac-proxy" containerID="cri-o://1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0" gracePeriod=30
Jul 09 18:11:21.324903 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.324877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:11:21.325284 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325267    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325287    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325301    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kube-rbac-proxy"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325310    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kube-rbac-proxy"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kube-rbac-proxy"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325335    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kube-rbac-proxy"
Jul 09 18:11:21.325395 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325396    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container"
Jul 09 18:11:21.325705 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325405    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container"
Jul 09 18:11:21.325705 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325511    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kserve-container"
Jul 09 18:11:21.325705 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325523    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kserve-container"
Jul 09 18:11:21.325705 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325537    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef22e18a-ec83-476f-b7ed-d4577bb5c748" containerName="kube-rbac-proxy"
Jul 09 18:11:21.325705 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.325546    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1326a459-a070-4137-9574-a50ba77b6ef9" containerName="kube-rbac-proxy"
Jul 09 18:11:21.334327 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.330158    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.334327 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.333520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5d567-predictor-serving-cert\""
Jul 09 18:11:21.334514 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.334413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5d567-kube-rbac-proxy-sar-config\""
Jul 09 18:11:21.338553 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.338532    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:11:21.420698 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.420666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:11:21.424213 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.424187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.427049 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.427007    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5d567-kube-rbac-proxy-sar-config\""
Jul 09 18:11:21.427173 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.427130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5d567-predictor-serving-cert\""
Jul 09 18:11:21.434197 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.434175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:11:21.434832 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.434811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.434938 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.434853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.434938 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.434876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsrw\" (UniqueName: \"kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.535781 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.535781 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.535781 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.536057 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4xs\" (UniqueName: \"kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.536057 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.536057 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.535900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsrw\" (UniqueName: \"kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.536555 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.536531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.538243 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.538217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.544011 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.543990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsrw\" (UniqueName: \"kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw\") pod \"success-200-isvc-5d567-predictor-69476f468b-r462g\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") " pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.636879 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.636834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.637070 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.636903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4xs\" (UniqueName: \"kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.637070 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.636950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.637505 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.637488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.639430 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.639411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.647679 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.647656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:21.648500 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.648482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4xs\" (UniqueName: \"kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs\") pod \"error-404-isvc-5d567-predictor-549f4c7f9-lbngd\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") " pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.735835 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.735802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:21.787326 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.786999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:11:21.790349 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:11:21.790184    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e60ae5_af8c_4efa_857f_d2998e757b5d.slice/crio-fda3a20c323e4404f96a896948da8f3f81322a2df05daf6d5158902a920fec38 WatchSource:0}: Error finding container fda3a20c323e4404f96a896948da8f3f81322a2df05daf6d5158902a920fec38: Status 404 returned error can't find the container with id fda3a20c323e4404f96a896948da8f3f81322a2df05daf6d5158902a920fec38
Jul 09 18:11:21.881751 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:21.881726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:11:21.890322 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:11:21.890290    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7ec986_945e_43cf_9881_67480013cb2c.slice/crio-bb7939f028af031e8d45ced2b48d6683e4e2b9a56bb3b11d0148a97a6621a858 WatchSource:0}: Error finding container bb7939f028af031e8d45ced2b48d6683e4e2b9a56bb3b11d0148a97a6621a858: Status 404 returned error can't find the container with id bb7939f028af031e8d45ced2b48d6683e4e2b9a56bb3b11d0148a97a6621a858
Jul 09 18:11:22.275664 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.275626    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerID="1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0" exitCode=2
Jul 09 18:11:22.276066 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.275701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerDied","Data":"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"}
Jul 09 18:11:22.277559 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.277533    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerID="f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be" exitCode=2
Jul 09 18:11:22.277687 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.277572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerDied","Data":"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"}
Jul 09 18:11:22.279473 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.279451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerStarted","Data":"512bfaec28bdd66486d2a78be3d9eea42030c160b3e88e3b4de38aa1105f33e0"}
Jul 09 18:11:22.279626 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.279611    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:22.279756 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.279733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerStarted","Data":"7834ba7d46f9d26f72b0e4879ca5ad76e7f06d4d4f7a07f763c5f46cfc35b41f"}
Jul 09 18:11:22.279888 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.279776    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:22.279888 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.279806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerStarted","Data":"fda3a20c323e4404f96a896948da8f3f81322a2df05daf6d5158902a920fec38"}
Jul 09 18:11:22.281154 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.281125    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:22.281516 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.281493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerStarted","Data":"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"}
Jul 09 18:11:22.281605 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.281544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerStarted","Data":"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"}
Jul 09 18:11:22.281605 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.281579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerStarted","Data":"bb7939f028af031e8d45ced2b48d6683e4e2b9a56bb3b11d0148a97a6621a858"}
Jul 09 18:11:22.281696 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.281617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:22.300399 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.300335    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podStartSLOduration=1.3003200750000001 podStartE2EDuration="1.300320075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:22.298814386 +0000 UTC m=+1895.901699843" watchObservedRunningTime="2026-07-09 18:11:22.300320075 +0000 UTC m=+1895.903205530"
Jul 09 18:11:22.317626 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:22.317581    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podStartSLOduration=1.317566331 podStartE2EDuration="1.317566331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:22.316457686 +0000 UTC m=+1895.919343146" watchObservedRunningTime="2026-07-09 18:11:22.317566331 +0000 UTC m=+1895.920451786"
Jul 09 18:11:23.289488 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:23.289445    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:23.290120 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:23.289642    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:23.290984 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:23.290951    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:23.766501 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:23.766463    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 09 18:11:23.771908 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:23.771884    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 09 18:11:24.295334 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.295285    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:24.736996 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.736972    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:11:24.739941 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.739923    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:11:24.765178 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config\") pod \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") "
Jul 09 18:11:24.765343 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls\") pod \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") "
Jul 09 18:11:24.765343 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765259    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smpnd\" (UniqueName: \"kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd\") pod \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") "
Jul 09 18:11:24.765343 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") pod \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") "
Jul 09 18:11:24.765343 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config\") pod \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\" (UID: \"6a8237ad-4fb6-443d-a9a7-54f8141e16b3\") "
Jul 09 18:11:24.765587 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlsbx\" (UniqueName: \"kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx\") pod \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\" (UID: \"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4\") "
Jul 09 18:11:24.765844 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a8237ad-4fb6-443d-a9a7-54f8141e16b3" (UID: "6a8237ad-4fb6-443d-a9a7-54f8141e16b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:24.765956 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.765882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" (UID: "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:24.767688 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.767664    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" (UID: "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:24.767914 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.767899    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd" (OuterVolumeSpecName: "kube-api-access-smpnd") pod "6a8237ad-4fb6-443d-a9a7-54f8141e16b3" (UID: "6a8237ad-4fb6-443d-a9a7-54f8141e16b3"). InnerVolumeSpecName "kube-api-access-smpnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:24.768001 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.767983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx" (OuterVolumeSpecName: "kube-api-access-vlsbx") pod "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" (UID: "a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4"). InnerVolumeSpecName "kube-api-access-vlsbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:24.768430 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.768334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a8237ad-4fb6-443d-a9a7-54f8141e16b3" (UID: "6a8237ad-4fb6-443d-a9a7-54f8141e16b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.866937    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.866965    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smpnd\" (UniqueName: \"kubernetes.io/projected/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-api-access-smpnd\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.866976    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.866984    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a8237ad-4fb6-443d-a9a7-54f8141e16b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.866993    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlsbx\" (UniqueName: \"kubernetes.io/projected/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-api-access-vlsbx\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:24.867022 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:24.867001    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:25.301372 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.301287    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerID="3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60" exitCode=0
Jul 09 18:11:25.301372 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.301339    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"
Jul 09 18:11:25.301897 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.301380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerDied","Data":"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"}
Jul 09 18:11:25.301897 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.301445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7" event={"ID":"6a8237ad-4fb6-443d-a9a7-54f8141e16b3","Type":"ContainerDied","Data":"1a7715ef898b7a56ed77f53b2c88e9a312d56a926bbd152d1eec2a103b4ac46d"}
Jul 09 18:11:25.301897 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.301470    2577 scope.go:117] "RemoveContainer" containerID="1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"
Jul 09 18:11:25.303244 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.303215    2577 generic.go:356] "Generic (PLEG): container finished" podID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerID="b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b" exitCode=0
Jul 09 18:11:25.303439 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.303263    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"
Jul 09 18:11:25.303439 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.303276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerDied","Data":"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"}
Jul 09 18:11:25.303439 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.303333    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx" event={"ID":"a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4","Type":"ContainerDied","Data":"b614710adb4297bebb6066b5a036e5a5cf03b24157b4bea20b792e4e7fd95755"}
Jul 09 18:11:25.310160 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.310145    2577 scope.go:117] "RemoveContainer" containerID="3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"
Jul 09 18:11:25.317443 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.317425    2577 scope.go:117] "RemoveContainer" containerID="1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"
Jul 09 18:11:25.317743 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:25.317714    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0\": container with ID starting with 1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0 not found: ID does not exist" containerID="1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"
Jul 09 18:11:25.317866 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.317754    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0"} err="failed to get container status \"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0\": rpc error: code = NotFound desc = could not find container \"1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0\": container with ID starting with 1fd1a3a1a6f132f16abc3085ea1b769bfd6a09d4a202e4e326aaef22ea4988a0 not found: ID does not exist"
Jul 09 18:11:25.317866 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.317777    2577 scope.go:117] "RemoveContainer" containerID="3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"
Jul 09 18:11:25.318294 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:25.318270    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60\": container with ID starting with 3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60 not found: ID does not exist" containerID="3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"
Jul 09 18:11:25.318407 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.318301    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60"} err="failed to get container status \"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60\": rpc error: code = NotFound desc = could not find container \"3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60\": container with ID starting with 3db87cc54ee895ccbb7b075e9d585e8929afb36c7fe770f52fe2a8bc57faca60 not found: ID does not exist"
Jul 09 18:11:25.318407 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.318323    2577 scope.go:117] "RemoveContainer" containerID="f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"
Jul 09 18:11:25.320587 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.320567    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:11:25.323599 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.323576    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67922-predictor-5948d8dd68-jknmx"]
Jul 09 18:11:25.326394 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.326349    2577 scope.go:117] "RemoveContainer" containerID="b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"
Jul 09 18:11:25.333317 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.333301    2577 scope.go:117] "RemoveContainer" containerID="f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"
Jul 09 18:11:25.333646 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:25.333623    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be\": container with ID starting with f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be not found: ID does not exist" containerID="f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"
Jul 09 18:11:25.333750 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.333653    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be"} err="failed to get container status \"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be\": rpc error: code = NotFound desc = could not find container \"f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be\": container with ID starting with f8a1068dd8573fecc24e8ba885278dbe666097a945acc08d893ba570ef6817be not found: ID does not exist"
Jul 09 18:11:25.333750 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.333668    2577 scope.go:117] "RemoveContainer" containerID="b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"
Jul 09 18:11:25.333919 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:25.333899    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b\": container with ID starting with b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b not found: ID does not exist" containerID="b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"
Jul 09 18:11:25.333981 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.333923    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b"} err="failed to get container status \"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b\": rpc error: code = NotFound desc = could not find container \"b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b\": container with ID starting with b13364015f266f5ac182ad3df14a4e5806824e32b3673f85545ba25c7c20b31b not found: ID does not exist"
Jul 09 18:11:25.334998 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.334978    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:11:25.339484 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:25.339464    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67922-predictor-5b8d4977c7-k7vt7"]
Jul 09 18:11:26.966642 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:26.966608    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" path="/var/lib/kubelet/pods/6a8237ad-4fb6-443d-a9a7-54f8141e16b3/volumes"
Jul 09 18:11:26.967036 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:26.967022    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" path="/var/lib/kubelet/pods/a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4/volumes"
Jul 09 18:11:28.295418 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:28.295391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:11:28.295889 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:28.295863    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:29.300876 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:29.300849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:11:29.301439 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:29.301389    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:31.385762 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.385722    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:11:31.386201 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.386024    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container" containerID="cri-o://857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52" gracePeriod=30
Jul 09 18:11:31.386201 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.386182    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kube-rbac-proxy" containerID="cri-o://85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8" gracePeriod=30
Jul 09 18:11:31.418870 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.418847    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:11:31.419220 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419207    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container"
Jul 09 18:11:31.419220 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419222    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419236    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419242    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419250    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419256    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419283    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419330 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419289    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419566 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419341    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419566 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419348    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kserve-container"
Jul 09 18:11:31.419566 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419371    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5bad1b4-d188-4ad0-9ef0-6b8ab06193d4" containerName="kube-rbac-proxy"
Jul 09 18:11:31.419566 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.419383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8237ad-4fb6-443d-a9a7-54f8141e16b3" containerName="kserve-container"
Jul 09 18:11:31.424306 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.424289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.426927 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.426908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-55ba8-kube-rbac-proxy-sar-config\""
Jul 09 18:11:31.427144 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.427131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-55ba8-predictor-serving-cert\""
Jul 09 18:11:31.432110 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.432090    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:11:31.492633 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.492591    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:11:31.492888 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.492859    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container" containerID="cri-o://834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0" gracePeriod=30
Jul 09 18:11:31.493180 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.492926    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kube-rbac-proxy" containerID="cri-o://545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7" gracePeriod=30
Jul 09 18:11:31.520704 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.520674    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:11:31.524069 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.524050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.524333 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.524313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglqz\" (UniqueName: \"kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.524462 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.524446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.524519 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.524474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.527233 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.527215    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-55ba8-predictor-serving-cert\""
Jul 09 18:11:31.527351 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.527233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-55ba8-kube-rbac-proxy-sar-config\""
Jul 09 18:11:31.531784 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.531764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:11:31.624870 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.624840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbsvg\" (UniqueName: \"kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.625009 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.624878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.625009 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.624896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.625009 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:31.624993    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-serving-cert: secret "success-200-isvc-55ba8-predictor-serving-cert" not found
Jul 09 18:11:31.625161 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.624987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglqz\" (UniqueName: \"kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.625161 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:31.625068    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls podName:da32fe16-b3e2-45c6-b20b-429e1eb9dabd nodeName:}" failed. No retries permitted until 2026-07-09 18:11:32.125047343 +0000 UTC m=+1905.727932797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls") pod "success-200-isvc-55ba8-predictor-65bd48468b-glbjd" (UID: "da32fe16-b3e2-45c6-b20b-429e1eb9dabd") : secret "success-200-isvc-55ba8-predictor-serving-cert" not found
Jul 09 18:11:31.625277 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.625177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.625277 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.625225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.625561 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.625545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.636486 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.636440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglqz\" (UniqueName: \"kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:31.726628 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.726598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.726780 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.726636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.726780 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.726659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbsvg\" (UniqueName: \"kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.726780 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:31.726773    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-serving-cert: secret "error-404-isvc-55ba8-predictor-serving-cert" not found
Jul 09 18:11:31.726913 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:31.726844    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls podName:811379ba-7a26-4cc2-ac2c-89d73fb5d560 nodeName:}" failed. No retries permitted until 2026-07-09 18:11:32.22681554 +0000 UTC m=+1905.829700987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls") pod "error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" (UID: "811379ba-7a26-4cc2-ac2c-89d73fb5d560") : secret "error-404-isvc-55ba8-predictor-serving-cert" not found
Jul 09 18:11:31.727228 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.727208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:31.737823 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:31.737801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbsvg\" (UniqueName: \"kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:32.131139 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.131095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:32.133452 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.133434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") pod \"success-200-isvc-55ba8-predictor-65bd48468b-glbjd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") " pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:32.232059 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.232021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:32.234299 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.234279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") pod \"error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") " pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:32.334914 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.334870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:32.338947 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.338925    2577 generic.go:356] "Generic (PLEG): container finished" podID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerID="545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7" exitCode=2
Jul 09 18:11:32.339027 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.339007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerDied","Data":"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"}
Jul 09 18:11:32.340690 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.340661    2577 generic.go:356] "Generic (PLEG): container finished" podID="daf37bea-70f1-4e36-9e83-a39881f93292" containerID="85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8" exitCode=2
Jul 09 18:11:32.340781 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.340701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerDied","Data":"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"}
Jul 09 18:11:32.434958 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.434922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:32.513608 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.512386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:11:32.588482 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.588452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:11:32.991280 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:32.991201    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 09 18:11:33.349779 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.349677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerStarted","Data":"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"}
Jul 09 18:11:33.349779 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.349752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerStarted","Data":"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"}
Jul 09 18:11:33.350017 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.349782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerStarted","Data":"6fd0451fa333178370cd8f8d7f599570b298b288ea24a7af3538942883ba5855"}
Jul 09 18:11:33.350017 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.349821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:33.350017 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.349849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:33.351687 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351657    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:11:33.351815 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerStarted","Data":"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"}
Jul 09 18:11:33.351815 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerStarted","Data":"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"}
Jul 09 18:11:33.351815 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerStarted","Data":"90ee12ea831301df83736bad6f2cf52378835840278e8d717dcefdc15c7bfe3e"}
Jul 09 18:11:33.351969 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:33.351969 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.351912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:33.352958 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.352932    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:11:33.368185 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.368143    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podStartSLOduration=2.368131471 podStartE2EDuration="2.368131471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:33.366760332 +0000 UTC m=+1906.969645816" watchObservedRunningTime="2026-07-09 18:11:33.368131471 +0000 UTC m=+1906.971016927"
Jul 09 18:11:33.998156 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:33.998120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 09 18:11:34.357005 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:34.356906    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:11:34.357158 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:34.357002    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:11:35.051207 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.051182    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:11:35.073678 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.073638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podStartSLOduration=4.07362267 podStartE2EDuration="4.07362267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:33.38879824 +0000 UTC m=+1906.991683700" watchObservedRunningTime="2026-07-09 18:11:35.07362267 +0000 UTC m=+1908.676508125"
Jul 09 18:11:35.157839 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.157811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t5sm\" (UniqueName: \"kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm\") pod \"4857bdc5-980d-47e0-aff9-fdd41039776b\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") "
Jul 09 18:11:35.157955 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.157881    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls\") pod \"4857bdc5-980d-47e0-aff9-fdd41039776b\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") "
Jul 09 18:11:35.157955 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.157951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config\") pod \"4857bdc5-980d-47e0-aff9-fdd41039776b\" (UID: \"4857bdc5-980d-47e0-aff9-fdd41039776b\") "
Jul 09 18:11:35.158420 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.158389    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4857bdc5-980d-47e0-aff9-fdd41039776b" (UID: "4857bdc5-980d-47e0-aff9-fdd41039776b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:35.161192 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.161160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4857bdc5-980d-47e0-aff9-fdd41039776b" (UID: "4857bdc5-980d-47e0-aff9-fdd41039776b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:35.161282 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.161215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm" (OuterVolumeSpecName: "kube-api-access-8t5sm") pod "4857bdc5-980d-47e0-aff9-fdd41039776b" (UID: "4857bdc5-980d-47e0-aff9-fdd41039776b"). InnerVolumeSpecName "kube-api-access-8t5sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:35.231547 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.231524    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:11:35.259271 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gncpb\" (UniqueName: \"kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb\") pod \"daf37bea-70f1-4e36-9e83-a39881f93292\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") "
Jul 09 18:11:35.259271 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config\") pod \"daf37bea-70f1-4e36-9e83-a39881f93292\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") "
Jul 09 18:11:35.259478 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls\") pod \"daf37bea-70f1-4e36-9e83-a39881f93292\" (UID: \"daf37bea-70f1-4e36-9e83-a39881f93292\") "
Jul 09 18:11:35.259478 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259444    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5sm\" (UniqueName: \"kubernetes.io/projected/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-api-access-8t5sm\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.259478 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259455    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4857bdc5-980d-47e0-aff9-fdd41039776b-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.259478 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259464    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4857bdc5-980d-47e0-aff9-fdd41039776b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.259693 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.259662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daf37bea-70f1-4e36-9e83-a39881f93292" (UID: "daf37bea-70f1-4e36-9e83-a39881f93292"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:11:35.261290 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.261265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daf37bea-70f1-4e36-9e83-a39881f93292" (UID: "daf37bea-70f1-4e36-9e83-a39881f93292"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:11:35.261372 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.261291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb" (OuterVolumeSpecName: "kube-api-access-gncpb") pod "daf37bea-70f1-4e36-9e83-a39881f93292" (UID: "daf37bea-70f1-4e36-9e83-a39881f93292"). InnerVolumeSpecName "kube-api-access-gncpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:11:35.360815 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.360789    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncpb\" (UniqueName: \"kubernetes.io/projected/daf37bea-70f1-4e36-9e83-a39881f93292-kube-api-access-gncpb\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.360815 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.360812    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daf37bea-70f1-4e36-9e83-a39881f93292-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.360985 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.360821    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daf37bea-70f1-4e36-9e83-a39881f93292-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:11:35.363694 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.363666    2577 generic.go:356] "Generic (PLEG): container finished" podID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerID="834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0" exitCode=0
Jul 09 18:11:35.363802 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.363717    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"
Jul 09 18:11:35.363802 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.363741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerDied","Data":"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"}
Jul 09 18:11:35.363889 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.363802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2" event={"ID":"4857bdc5-980d-47e0-aff9-fdd41039776b","Type":"ContainerDied","Data":"5683f259cfabdf7eeef31fdecb7369cb087918376db0d3b48f3684abfa293810"}
Jul 09 18:11:35.363889 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.363839    2577 scope.go:117] "RemoveContainer" containerID="545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"
Jul 09 18:11:35.365426 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.365399    2577 generic.go:356] "Generic (PLEG): container finished" podID="daf37bea-70f1-4e36-9e83-a39881f93292" containerID="857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52" exitCode=0
Jul 09 18:11:35.365563 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.365442    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"
Jul 09 18:11:35.365563 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.365483    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerDied","Data":"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"}
Jul 09 18:11:35.365682 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.365589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn" event={"ID":"daf37bea-70f1-4e36-9e83-a39881f93292","Type":"ContainerDied","Data":"6faf8e17419b99501cdc786230fb2e1b84195491d9f58523cfe377424e79d6db"}
Jul 09 18:11:35.373194 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.373172    2577 scope.go:117] "RemoveContainer" containerID="834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"
Jul 09 18:11:35.380867 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.380851    2577 scope.go:117] "RemoveContainer" containerID="545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"
Jul 09 18:11:35.381123 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:35.381105    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7\": container with ID starting with 545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7 not found: ID does not exist" containerID="545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"
Jul 09 18:11:35.381184 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.381130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7"} err="failed to get container status \"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7\": rpc error: code = NotFound desc = could not find container \"545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7\": container with ID starting with 545d0ffa20749ffa2fb5c976754e3f48baab92e8b7abba4d537dd11881381dd7 not found: ID does not exist"
Jul 09 18:11:35.381184 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.381146    2577 scope.go:117] "RemoveContainer" containerID="834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"
Jul 09 18:11:35.381394 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:35.381373    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0\": container with ID starting with 834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0 not found: ID does not exist" containerID="834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"
Jul 09 18:11:35.381441 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.381402    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0"} err="failed to get container status \"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0\": rpc error: code = NotFound desc = could not find container \"834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0\": container with ID starting with 834e669073fb4808850851d01eda7172d6d11f8c6d1613bf55b76dc526c36dc0 not found: ID does not exist"
Jul 09 18:11:35.381441 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.381417    2577 scope.go:117] "RemoveContainer" containerID="85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"
Jul 09 18:11:35.393333 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.393312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:11:35.397337 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.397318    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-07956-predictor-7d74d5c7dc-zctn2"]
Jul 09 18:11:35.397872 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.397857    2577 scope.go:117] "RemoveContainer" containerID="857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"
Jul 09 18:11:35.404939 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.404924    2577 scope.go:117] "RemoveContainer" containerID="85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"
Jul 09 18:11:35.405193 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:35.405174    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8\": container with ID starting with 85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8 not found: ID does not exist" containerID="85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"
Jul 09 18:11:35.405253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.405198    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8"} err="failed to get container status \"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8\": rpc error: code = NotFound desc = could not find container \"85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8\": container with ID starting with 85628b20733a3c954036d5f53d79ecacb2a9f17d2b6f9eb487953082302c29a8 not found: ID does not exist"
Jul 09 18:11:35.405253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.405219    2577 scope.go:117] "RemoveContainer" containerID="857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"
Jul 09 18:11:35.405488 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:11:35.405470    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52\": container with ID starting with 857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52 not found: ID does not exist" containerID="857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"
Jul 09 18:11:35.405539 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.405493    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52"} err="failed to get container status \"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52\": rpc error: code = NotFound desc = could not find container \"857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52\": container with ID starting with 857e653c89396408c329a3a6808c92a00d231c48c37df0f6d413e966191a8d52 not found: ID does not exist"
Jul 09 18:11:35.409009 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.408990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:11:35.412749 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:35.412732    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-07956-predictor-649f94cfd6-mgsrn"]
Jul 09 18:11:36.966232 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:36.966203    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" path="/var/lib/kubelet/pods/4857bdc5-980d-47e0-aff9-fdd41039776b/volumes"
Jul 09 18:11:36.966635 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:36.966619    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" path="/var/lib/kubelet/pods/daf37bea-70f1-4e36-9e83-a39881f93292/volumes"
Jul 09 18:11:38.296571 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:38.296531    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:39.302302 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:39.302260    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:39.361255 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:39.361230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:11:39.361612 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:39.361592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:11:39.361727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:39.361640    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:11:39.361999 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:39.361976    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:11:48.296790 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:48.296750    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:49.301932 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:49.301895    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:49.362354 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:49.362316    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:11:49.362521 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:49.362317    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:11:58.295933 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:58.295892    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:11:59.301954 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:59.301911    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:11:59.362515 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:59.362477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:11:59.362668 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:11:59.362477    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:12:08.296774 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:08.296735    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 09 18:12:09.301502 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:09.301459    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 09 18:12:09.361717 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:09.361671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 09 18:12:09.362135 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:09.362108    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:12:18.297197 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:18.297160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:12:19.302495 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:19.302465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:12:19.362566 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:19.362521    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 09 18:12:19.363399 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:19.363379    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:12:29.362533 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:12:29.362500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:14:48.453894 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:14:48.453868    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:14:48.456978 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:14:48.456957    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:19:48.480844 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:19:48.480713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:19:48.485764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:19:48.485745    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:20:46.075647 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.075614    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:20:46.076251 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.075827    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container" containerID="cri-o://f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515" gracePeriod=30
Jul 09 18:20:46.076251 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.075867    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kube-rbac-proxy" containerID="cri-o://5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da" gracePeriod=30
Jul 09 18:20:46.124927 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.124900    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:20:46.125133 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.125110    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container" containerID="cri-o://3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294" gracePeriod=30
Jul 09 18:20:46.125212 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.125145    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kube-rbac-proxy" containerID="cri-o://ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746" gracePeriod=30
Jul 09 18:20:46.776752 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.776721    2577 generic.go:356] "Generic (PLEG): container finished" podID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerID="5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da" exitCode=2
Jul 09 18:20:46.776934 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.776799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerDied","Data":"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"}
Jul 09 18:20:46.778392 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.778347    2577 generic.go:356] "Generic (PLEG): container finished" podID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerID="ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746" exitCode=2
Jul 09 18:20:46.778506 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:46.778400    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerDied","Data":"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"}
Jul 09 18:20:49.247409 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.247338    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:20:49.280063 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.280036    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:20:49.312598 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.312576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cglqz\" (UniqueName: \"kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz\") pod \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") "
Jul 09 18:20:49.312708 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.312662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") pod \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") "
Jul 09 18:20:49.312708 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.312698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config\") pod \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\" (UID: \"da32fe16-b3e2-45c6-b20b-429e1eb9dabd\") "
Jul 09 18:20:49.313054 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.313034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da32fe16-b3e2-45c6-b20b-429e1eb9dabd" (UID: "da32fe16-b3e2-45c6-b20b-429e1eb9dabd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:20:49.314626 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.314605    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz" (OuterVolumeSpecName: "kube-api-access-cglqz") pod "da32fe16-b3e2-45c6-b20b-429e1eb9dabd" (UID: "da32fe16-b3e2-45c6-b20b-429e1eb9dabd"). InnerVolumeSpecName "kube-api-access-cglqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:49.314711 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.314623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da32fe16-b3e2-45c6-b20b-429e1eb9dabd" (UID: "da32fe16-b3e2-45c6-b20b-429e1eb9dabd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:49.413500 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbsvg\" (UniqueName: \"kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg\") pod \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") "
Jul 09 18:20:49.413603 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config\") pod \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") "
Jul 09 18:20:49.413603 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413569    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") pod \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\" (UID: \"811379ba-7a26-4cc2-ac2c-89d73fb5d560\") "
Jul 09 18:20:49.413787 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413763    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.413830 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413792    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cglqz\" (UniqueName: \"kubernetes.io/projected/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-kube-api-access-cglqz\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.413830 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413809    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da32fe16-b3e2-45c6-b20b-429e1eb9dabd-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.413927 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.413898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "811379ba-7a26-4cc2-ac2c-89d73fb5d560" (UID: "811379ba-7a26-4cc2-ac2c-89d73fb5d560"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:20:49.415483 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.415457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg" (OuterVolumeSpecName: "kube-api-access-kbsvg") pod "811379ba-7a26-4cc2-ac2c-89d73fb5d560" (UID: "811379ba-7a26-4cc2-ac2c-89d73fb5d560"). InnerVolumeSpecName "kube-api-access-kbsvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:49.415567 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.415466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "811379ba-7a26-4cc2-ac2c-89d73fb5d560" (UID: "811379ba-7a26-4cc2-ac2c-89d73fb5d560"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:49.514963 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.514906    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbsvg\" (UniqueName: \"kubernetes.io/projected/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-api-access-kbsvg\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.514963 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.514928    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/811379ba-7a26-4cc2-ac2c-89d73fb5d560-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.514963 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.514937    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/811379ba-7a26-4cc2-ac2c-89d73fb5d560-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:20:49.793002 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.792921    2577 generic.go:356] "Generic (PLEG): container finished" podID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerID="f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515" exitCode=0
Jul 09 18:20:49.793002 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.792968    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"
Jul 09 18:20:49.793180 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.793006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerDied","Data":"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"}
Jul 09 18:20:49.793180 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.793060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd" event={"ID":"da32fe16-b3e2-45c6-b20b-429e1eb9dabd","Type":"ContainerDied","Data":"90ee12ea831301df83736bad6f2cf52378835840278e8d717dcefdc15c7bfe3e"}
Jul 09 18:20:49.793180 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.793094    2577 scope.go:117] "RemoveContainer" containerID="5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"
Jul 09 18:20:49.794737 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.794714    2577 generic.go:356] "Generic (PLEG): container finished" podID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerID="3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294" exitCode=0
Jul 09 18:20:49.794877 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.794743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerDied","Data":"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"}
Jul 09 18:20:49.794877 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.794779    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"
Jul 09 18:20:49.794877 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.794790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg" event={"ID":"811379ba-7a26-4cc2-ac2c-89d73fb5d560","Type":"ContainerDied","Data":"6fd0451fa333178370cd8f8d7f599570b298b288ea24a7af3538942883ba5855"}
Jul 09 18:20:49.803479 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.803460    2577 scope.go:117] "RemoveContainer" containerID="f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"
Jul 09 18:20:49.811100 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.811086    2577 scope.go:117] "RemoveContainer" containerID="5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"
Jul 09 18:20:49.811351 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:20:49.811334    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da\": container with ID starting with 5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da not found: ID does not exist" containerID="5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"
Jul 09 18:20:49.811531 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.811377    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da"} err="failed to get container status \"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da\": rpc error: code = NotFound desc = could not find container \"5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da\": container with ID starting with 5a1975ac0bbc7cf7c1b37efc0170e8170cd736491b97a7f8aa6d93ea640996da not found: ID does not exist"
Jul 09 18:20:49.811531 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.811394    2577 scope.go:117] "RemoveContainer" containerID="f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"
Jul 09 18:20:49.811625 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:20:49.811592    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515\": container with ID starting with f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515 not found: ID does not exist" containerID="f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"
Jul 09 18:20:49.811625 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.811607    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515"} err="failed to get container status \"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515\": rpc error: code = NotFound desc = could not find container \"f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515\": container with ID starting with f3648a92df052951580e04fe7b149d6df4fc9d804842982f72be86dab9584515 not found: ID does not exist"
Jul 09 18:20:49.811625 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.811618    2577 scope.go:117] "RemoveContainer" containerID="ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"
Jul 09 18:20:49.816336 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.816312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:20:49.819134 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.819115    2577 scope.go:117] "RemoveContainer" containerID="3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"
Jul 09 18:20:49.820633 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.820616    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-55ba8-predictor-65bd48468b-glbjd"]
Jul 09 18:20:49.826451 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.826432    2577 scope.go:117] "RemoveContainer" containerID="ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"
Jul 09 18:20:49.826718 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:20:49.826700    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746\": container with ID starting with ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746 not found: ID does not exist" containerID="ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"
Jul 09 18:20:49.826784 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.826724    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746"} err="failed to get container status \"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746\": rpc error: code = NotFound desc = could not find container \"ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746\": container with ID starting with ea4a0ebea70055f10a40174ddafc27c5b9ae8456912d01aaa6d291e1a2c35746 not found: ID does not exist"
Jul 09 18:20:49.826784 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.826741    2577 scope.go:117] "RemoveContainer" containerID="3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"
Jul 09 18:20:49.826938 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:20:49.826921    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294\": container with ID starting with 3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294 not found: ID does not exist" containerID="3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"
Jul 09 18:20:49.826981 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.826944    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294"} err="failed to get container status \"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294\": rpc error: code = NotFound desc = could not find container \"3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294\": container with ID starting with 3f69e55936ed2bf0086fa52925670877279dff1a9e6c6f2948580cc247489294 not found: ID does not exist"
Jul 09 18:20:49.831071 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.831052    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:20:49.836783 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:49.836764    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55ba8-predictor-7c8f5d49f9-wmxwg"]
Jul 09 18:20:50.967080 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:50.967046    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" path="/var/lib/kubelet/pods/811379ba-7a26-4cc2-ac2c-89d73fb5d560/volumes"
Jul 09 18:20:50.967668 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:20:50.967646    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" path="/var/lib/kubelet/pods/da32fe16-b3e2-45c6-b20b-429e1eb9dabd/volumes"
Jul 09 18:24:48.507024 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:24:48.506899    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:24:48.512798 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:24:48.512779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:28:50.617407 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.617351    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:28:50.617956 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.617650    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container" containerID="cri-o://7834ba7d46f9d26f72b0e4879ca5ad76e7f06d4d4f7a07f763c5f46cfc35b41f" gracePeriod=30
Jul 09 18:28:50.617956 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.617698    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kube-rbac-proxy" containerID="cri-o://512bfaec28bdd66486d2a78be3d9eea42030c160b3e88e3b4de38aa1105f33e0" gracePeriod=30
Jul 09 18:28:50.679639 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.679607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:28:50.679947 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.679879    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container" containerID="cri-o://4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114" gracePeriod=30
Jul 09 18:28:50.680034 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.679993    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kube-rbac-proxy" containerID="cri-o://3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340" gracePeriod=30
Jul 09 18:28:50.745855 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.745823    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerID="512bfaec28bdd66486d2a78be3d9eea42030c160b3e88e3b4de38aa1105f33e0" exitCode=2
Jul 09 18:28:50.746006 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:50.745859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerDied","Data":"512bfaec28bdd66486d2a78be3d9eea42030c160b3e88e3b4de38aa1105f33e0"}
Jul 09 18:28:51.751942 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:51.751908    2577 generic.go:356] "Generic (PLEG): container finished" podID="be7ec986-945e-43cf-9881-67480013cb2c" containerID="3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340" exitCode=2
Jul 09 18:28:51.752342 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:51.751974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerDied","Data":"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"}
Jul 09 18:28:53.290322 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:53.290267    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 09 18:28:53.765306 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:53.765271    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerID="7834ba7d46f9d26f72b0e4879ca5ad76e7f06d4d4f7a07f763c5f46cfc35b41f" exitCode=0
Jul 09 18:28:53.765487 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:53.765338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerDied","Data":"7834ba7d46f9d26f72b0e4879ca5ad76e7f06d4d4f7a07f763c5f46cfc35b41f"}
Jul 09 18:28:53.904293 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:53.904267    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:28:54.036418 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.036387    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:28:54.084093 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.084054    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnsrw\" (UniqueName: \"kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw\") pod \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") "
Jul 09 18:28:54.084281 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.084156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config\") pod \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") "
Jul 09 18:28:54.084281 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.084258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls\") pod \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\" (UID: \"a6e60ae5-af8c-4efa-857f-d2998e757b5d\") "
Jul 09 18:28:54.084493 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.084471    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6e60ae5-af8c-4efa-857f-d2998e757b5d" (UID: "a6e60ae5-af8c-4efa-857f-d2998e757b5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:54.086158 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.086131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw" (OuterVolumeSpecName: "kube-api-access-fnsrw") pod "a6e60ae5-af8c-4efa-857f-d2998e757b5d" (UID: "a6e60ae5-af8c-4efa-857f-d2998e757b5d"). InnerVolumeSpecName "kube-api-access-fnsrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:54.086262 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.086178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6e60ae5-af8c-4efa-857f-d2998e757b5d" (UID: "a6e60ae5-af8c-4efa-857f-d2998e757b5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:54.184984 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.184951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls\") pod \"be7ec986-945e-43cf-9881-67480013cb2c\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") "
Jul 09 18:28:54.184984 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.184991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config\") pod \"be7ec986-945e-43cf-9881-67480013cb2c\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") "
Jul 09 18:28:54.185253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.185022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4xs\" (UniqueName: \"kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs\") pod \"be7ec986-945e-43cf-9881-67480013cb2c\" (UID: \"be7ec986-945e-43cf-9881-67480013cb2c\") "
Jul 09 18:28:54.185253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.185169    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6e60ae5-af8c-4efa-857f-d2998e757b5d-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.185253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.185180    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsrw\" (UniqueName: \"kubernetes.io/projected/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-api-access-fnsrw\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.185253 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.185188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6e60ae5-af8c-4efa-857f-d2998e757b5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.185525 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.185349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be7ec986-945e-43cf-9881-67480013cb2c" (UID: "be7ec986-945e-43cf-9881-67480013cb2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:28:54.187072 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.187048    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be7ec986-945e-43cf-9881-67480013cb2c" (UID: "be7ec986-945e-43cf-9881-67480013cb2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:28:54.187157 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.187095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs" (OuterVolumeSpecName: "kube-api-access-kb4xs") pod "be7ec986-945e-43cf-9881-67480013cb2c" (UID: "be7ec986-945e-43cf-9881-67480013cb2c"). InnerVolumeSpecName "kube-api-access-kb4xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:28:54.286256 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.286160    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be7ec986-945e-43cf-9881-67480013cb2c-proxy-tls\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.286256 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.286192    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be7ec986-945e-43cf-9881-67480013cb2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.286256 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.286205    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4xs\" (UniqueName: \"kubernetes.io/projected/be7ec986-945e-43cf-9881-67480013cb2c-kube-api-access-kb4xs\") on node \"ip-10-0-136-100.ec2.internal\" DevicePath \"\""
Jul 09 18:28:54.771067 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.771036    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"
Jul 09 18:28:54.771533 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.771076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g" event={"ID":"a6e60ae5-af8c-4efa-857f-d2998e757b5d","Type":"ContainerDied","Data":"fda3a20c323e4404f96a896948da8f3f81322a2df05daf6d5158902a920fec38"}
Jul 09 18:28:54.771533 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.771150    2577 scope.go:117] "RemoveContainer" containerID="512bfaec28bdd66486d2a78be3d9eea42030c160b3e88e3b4de38aa1105f33e0"
Jul 09 18:28:54.772827 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.772807    2577 generic.go:356] "Generic (PLEG): container finished" podID="be7ec986-945e-43cf-9881-67480013cb2c" containerID="4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114" exitCode=0
Jul 09 18:28:54.772910 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.772869    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"
Jul 09 18:28:54.772960 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.772878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerDied","Data":"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"}
Jul 09 18:28:54.773001 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.772986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd" event={"ID":"be7ec986-945e-43cf-9881-67480013cb2c","Type":"ContainerDied","Data":"bb7939f028af031e8d45ced2b48d6683e4e2b9a56bb3b11d0148a97a6621a858"}
Jul 09 18:28:54.781474 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.781453    2577 scope.go:117] "RemoveContainer" containerID="7834ba7d46f9d26f72b0e4879ca5ad76e7f06d4d4f7a07f763c5f46cfc35b41f"
Jul 09 18:28:54.789308 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.789290    2577 scope.go:117] "RemoveContainer" containerID="3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"
Jul 09 18:28:54.796843 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.796826    2577 scope.go:117] "RemoveContainer" containerID="4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"
Jul 09 18:28:54.797978 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.797959    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:28:54.801181 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.801157    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5d567-predictor-549f4c7f9-lbngd"]
Jul 09 18:28:54.804600 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.804584    2577 scope.go:117] "RemoveContainer" containerID="3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"
Jul 09 18:28:54.804882 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:28:54.804863    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340\": container with ID starting with 3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340 not found: ID does not exist" containerID="3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"
Jul 09 18:28:54.804950 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.804888    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340"} err="failed to get container status \"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340\": rpc error: code = NotFound desc = could not find container \"3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340\": container with ID starting with 3d4885da0822f972b87d96878b6648d53eb35af6960a127283a4f90cd0a48340 not found: ID does not exist"
Jul 09 18:28:54.804950 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.804905    2577 scope.go:117] "RemoveContainer" containerID="4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"
Jul 09 18:28:54.805123 ip-10-0-136-100 kubenswrapper[2577]: E0709 18:28:54.805106    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114\": container with ID starting with 4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114 not found: ID does not exist" containerID="4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"
Jul 09 18:28:54.805164 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.805130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114"} err="failed to get container status \"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114\": rpc error: code = NotFound desc = could not find container \"4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114\": container with ID starting with 4b46f1d75d92df401eb562e39b52201a96c2daddec218f2a11a3a2f929dea114 not found: ID does not exist"
Jul 09 18:28:54.812894 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.812864    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:28:54.819290 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.819269    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5d567-predictor-69476f468b-r462g"]
Jul 09 18:28:54.966390 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.966291    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" path="/var/lib/kubelet/pods/a6e60ae5-af8c-4efa-857f-d2998e757b5d/volumes"
Jul 09 18:28:54.966738 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:28:54.966723    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be7ec986-945e-43cf-9881-67480013cb2c" path="/var/lib/kubelet/pods/be7ec986-945e-43cf-9881-67480013cb2c/volumes"
Jul 09 18:29:23.380264 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380187    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q7djf/must-gather-fgqtd"]
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380513    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380525    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380533    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380539    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380548    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380553    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380558    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380564    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380570    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380576    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380584    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380589    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380595    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380599    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380607    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380622    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380627    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380637    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380642    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380648    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380653    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380664    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380714    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380722    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kube-rbac-proxy"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380729    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380735    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4857bdc5-980d-47e0-aff9-fdd41039776b" containerName="kserve-container"
Jul 09 18:29:23.380727 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380741    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kube-rbac-proxy"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380748    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kube-rbac-proxy"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380753    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kube-rbac-proxy"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380758    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be7ec986-945e-43cf-9881-67480013cb2c" containerName="kserve-container"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380763    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="811379ba-7a26-4cc2-ac2c-89d73fb5d560" containerName="kserve-container"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380768    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6e60ae5-af8c-4efa-857f-d2998e757b5d" containerName="kube-rbac-proxy"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380773    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da32fe16-b3e2-45c6-b20b-429e1eb9dabd" containerName="kserve-container"
Jul 09 18:29:23.381629 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.380778    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf37bea-70f1-4e36-9e83-a39881f93292" containerName="kserve-container"
Jul 09 18:29:23.383765 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.383749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.387121 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.387101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q7djf\"/\"openshift-service-ca.crt\""
Jul 09 18:29:23.387218 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.387128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q7djf\"/\"kube-root-ca.crt\""
Jul 09 18:29:23.388590 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.388571    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q7djf\"/\"default-dockercfg-n8q72\""
Jul 09 18:29:23.393251 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.393232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/must-gather-fgqtd"]
Jul 09 18:29:23.419440 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.419420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxgh\" (UniqueName: \"kubernetes.io/projected/b575e9c3-2cbd-47ca-a37a-e234ce09753f-kube-api-access-rgxgh\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.419545 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.419467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b575e9c3-2cbd-47ca-a37a-e234ce09753f-must-gather-output\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.520765 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.520736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxgh\" (UniqueName: \"kubernetes.io/projected/b575e9c3-2cbd-47ca-a37a-e234ce09753f-kube-api-access-rgxgh\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.520881 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.520780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b575e9c3-2cbd-47ca-a37a-e234ce09753f-must-gather-output\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.521068 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.521053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b575e9c3-2cbd-47ca-a37a-e234ce09753f-must-gather-output\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.529281 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.529252    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxgh\" (UniqueName: \"kubernetes.io/projected/b575e9c3-2cbd-47ca-a37a-e234ce09753f-kube-api-access-rgxgh\") pod \"must-gather-fgqtd\" (UID: \"b575e9c3-2cbd-47ca-a37a-e234ce09753f\") " pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.701198 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.701173    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/must-gather-fgqtd"
Jul 09 18:29:23.830176 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.830144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/must-gather-fgqtd"]
Jul 09 18:29:23.833864 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:29:23.833843    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb575e9c3_2cbd_47ca_a37a_e234ce09753f.slice/crio-61831ce429b104eafb4b8a124fb7e4eedd3048dc965fa6d3d9eaf38867aa9072 WatchSource:0}: Error finding container 61831ce429b104eafb4b8a124fb7e4eedd3048dc965fa6d3d9eaf38867aa9072: Status 404 returned error can't find the container with id 61831ce429b104eafb4b8a124fb7e4eedd3048dc965fa6d3d9eaf38867aa9072
Jul 09 18:29:23.835983 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.835967    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:29:23.893065 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:23.893041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/must-gather-fgqtd" event={"ID":"b575e9c3-2cbd-47ca-a37a-e234ce09753f","Type":"ContainerStarted","Data":"61831ce429b104eafb4b8a124fb7e4eedd3048dc965fa6d3d9eaf38867aa9072"}
Jul 09 18:29:24.909825 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:24.909768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/must-gather-fgqtd" event={"ID":"b575e9c3-2cbd-47ca-a37a-e234ce09753f","Type":"ContainerStarted","Data":"243a28820f1cd41da6f623d6975bfdd2493e315af8306c0f9d8f241c5f37608e"}
Jul 09 18:29:24.910346 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:24.909837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/must-gather-fgqtd" event={"ID":"b575e9c3-2cbd-47ca-a37a-e234ce09753f","Type":"ContainerStarted","Data":"1e4186b916848d62fcae2b78e57ca566feaf75cd3b9dccd7a579841689e1cb8c"}
Jul 09 18:29:24.933894 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:24.933837    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q7djf/must-gather-fgqtd" podStartSLOduration=1.341028726 podStartE2EDuration="1.933818776s" totalImagesPullingTime="592.79005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:24.932598006 +0000 UTC m=+2978.535483456" watchObservedRunningTime="2026-07-09 18:29:24.933818776 +0000 UTC m=+2978.536704234"
Jul 09 18:29:25.934048 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:25.934022    2577 ???:1] "http: TLS handshake error from 10.0.141.169:33340: EOF"
Jul 09 18:29:25.946670 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:25.946641    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ss8kv_76cd4c1f-fee0-4f8c-b9ac-026346a096bb/global-pull-secret-syncer/0.log"
Jul 09 18:29:26.079785 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:26.079748    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xvfv6_13d0e87c-b00d-40ec-81cc-95e594606af8/konnectivity-agent/0.log"
Jul 09 18:29:26.161226 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:26.161198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-100.ec2.internal_40a25d067cd687c255b64734b988a9bd/haproxy/0.log"
Jul 09 18:29:30.271749 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.270191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54b878b4fd-7nlvt_f2372a27-0aaf-457e-81be-d46a7acadaba/metrics-server/0.log"
Jul 09 18:29:30.426006 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.425974    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8zcp_9c1fd3e8-f2b9-405e-9b65-19fde324d14f/node-exporter/0.log"
Jul 09 18:29:30.457045 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.457005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8zcp_9c1fd3e8-f2b9-405e-9b65-19fde324d14f/kube-rbac-proxy/0.log"
Jul 09 18:29:30.485725 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.485698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8zcp_9c1fd3e8-f2b9-405e-9b65-19fde324d14f/init-textfile/0.log"
Jul 09 18:29:30.630578 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.630502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jhrdz_596413e4-cb93-4eb1-b1c4-9fde83ed4841/kube-rbac-proxy-main/0.log"
Jul 09 18:29:30.660091 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.660064    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jhrdz_596413e4-cb93-4eb1-b1c4-9fde83ed4841/kube-rbac-proxy-self/0.log"
Jul 09 18:29:30.685633 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:30.685604    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-jhrdz_596413e4-cb93-4eb1-b1c4-9fde83ed4841/openshift-state-metrics/0.log"
Jul 09 18:29:31.093582 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.093477    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/thanos-query/0.log"
Jul 09 18:29:31.118163 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.118131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/kube-rbac-proxy-web/0.log"
Jul 09 18:29:31.144814 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.144779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/kube-rbac-proxy/0.log"
Jul 09 18:29:31.171601 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.171561    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/prom-label-proxy/0.log"
Jul 09 18:29:31.201717 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.201680    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/kube-rbac-proxy-rules/0.log"
Jul 09 18:29:31.238792 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:31.238763    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cdd574c69-7wwk5_678086a1-b9f4-48d0-a9fd-02a9e943551f/kube-rbac-proxy-metrics/0.log"
Jul 09 18:29:32.536213 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:32.536183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-wklcb_e04c5bee-815a-458c-a969-5a7cdbab06be/networking-console-plugin/0.log"
Jul 09 18:29:33.586103 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:33.586075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-ls8mz_37a82a7b-936f-4f81-943d-a1fcc555143a/download-server/0.log"
Jul 09 18:29:34.198672 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.198637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"]
Jul 09 18:29:34.204602 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.204581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.211198 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.211108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"]
Jul 09 18:29:34.323449 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.323419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmsp\" (UniqueName: \"kubernetes.io/projected/82183730-c88f-422a-aaaf-40b4c5ad94bb-kube-api-access-znmsp\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.323620 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.323491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-sys\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.323620 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.323515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-podres\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.323620 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.323532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-proc\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.323620 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.323565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-lib-modules\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424475 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmsp\" (UniqueName: \"kubernetes.io/projected/82183730-c88f-422a-aaaf-40b4c5ad94bb-kube-api-access-znmsp\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424668 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-sys\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424668 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-podres\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424796 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-sys\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424796 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-proc\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424796 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-podres\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424796 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-lib-modules\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424944 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-proc\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.424944 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.424887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82183730-c88f-422a-aaaf-40b4c5ad94bb-lib-modules\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.435057 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.435035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmsp\" (UniqueName: \"kubernetes.io/projected/82183730-c88f-422a-aaaf-40b4c5ad94bb-kube-api-access-znmsp\") pod \"perf-node-gather-daemonset-5hrjl\" (UID: \"82183730-c88f-422a-aaaf-40b4c5ad94bb\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.521909 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.521830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.684133 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.684104    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"]
Jul 09 18:29:34.687779 ip-10-0-136-100 kubenswrapper[2577]: W0709 18:29:34.687737    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82183730_c88f_422a_aaaf_40b4c5ad94bb.slice/crio-0818b321abe7d5875566df0d80f68d3f5616d7cd354acf4e06bc3834b46bc595 WatchSource:0}: Error finding container 0818b321abe7d5875566df0d80f68d3f5616d7cd354acf4e06bc3834b46bc595: Status 404 returned error can't find the container with id 0818b321abe7d5875566df0d80f68d3f5616d7cd354acf4e06bc3834b46bc595
Jul 09 18:29:34.970465 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.970430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl" event={"ID":"82183730-c88f-422a-aaaf-40b4c5ad94bb","Type":"ContainerStarted","Data":"f006f41506dfb1c33a8ee6e075aadac538fd3626e3f2c780878726e62b1b8417"}
Jul 09 18:29:34.970633 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.970487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl" event={"ID":"82183730-c88f-422a-aaaf-40b4c5ad94bb","Type":"ContainerStarted","Data":"0818b321abe7d5875566df0d80f68d3f5616d7cd354acf4e06bc3834b46bc595"}
Jul 09 18:29:34.970633 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.970609    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:34.991673 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:34.989249    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl" podStartSLOduration=0.98922926 podStartE2EDuration="989.22926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:34.988558382 +0000 UTC m=+2988.591443840" watchObservedRunningTime="2026-07-09 18:29:34.98922926 +0000 UTC m=+2988.592114717"
Jul 09 18:29:35.152320 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:35.152241    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vc6k_f34e57e5-2188-471a-91ff-92b558e41ea6/dns/0.log"
Jul 09 18:29:35.174376 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:35.174339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vc6k_f34e57e5-2188-471a-91ff-92b558e41ea6/kube-rbac-proxy/0.log"
Jul 09 18:29:35.253764 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:35.253739    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg4dz_2d7d8c3a-52bd-4089-8af7-90a74d3abfab/dns-node-resolver/0.log"
Jul 09 18:29:35.820704 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:35.820671    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-784764f4c7-kjnbw_d4a1675f-4846-4a5b-a92e-b61241620048/registry/0.log"
Jul 09 18:29:35.867835 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:35.867805    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-757rc_6a13b98e-d5f8-46db-8552-838b46760281/node-ca/0.log"
Jul 09 18:29:36.737348 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:36.737317    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7945b9f8cd-6dlzr_b0fe08a2-2320-46a9-8b03-08f8af1a8c67/router/0.log"
Jul 09 18:29:37.082057 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.081978    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hr5d_c910f899-63b6-466d-adaa-b66ed83517fa/serve-healthcheck-canary/0.log"
Jul 09 18:29:37.508037 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.508007    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-2xvxr_5ef9402b-6459-4239-ad0d-0e5d825bd721/insights-operator/0.log"
Jul 09 18:29:37.509523 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.509502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-2xvxr_5ef9402b-6459-4239-ad0d-0e5d825bd721/insights-operator/1.log"
Jul 09 18:29:37.532101 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.532085    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-828zp_23b09a43-b99a-40dd-b630-b4e109562963/kube-rbac-proxy/0.log"
Jul 09 18:29:37.553548 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.553532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-828zp_23b09a43-b99a-40dd-b630-b4e109562963/exporter/0.log"
Jul 09 18:29:37.575241 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:37.575216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-828zp_23b09a43-b99a-40dd-b630-b4e109562963/extractor/0.log"
Jul 09 18:29:39.814619 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:39.814591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-2tmt4_0eb0906c-1980-4618-9b23-ff3f1d5e49dd/server/0.log"
Jul 09 18:29:39.984850 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:39.984814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-m22w5_905c8dc9-6851-4a0d-b66b-5b6da1131e74/manager/0.log"
Jul 09 18:29:40.078599 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:40.078525    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-htvdb_2fae0562-3b14-4677-9cb3-deaf67baa603/seaweedfs/0.log"
Jul 09 18:29:40.985742 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:40.985713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-5hrjl"
Jul 09 18:29:44.120507 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:44.120470    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-sgt5h_9bdd5402-1bab-497d-a86b-d5d5f1543a51/migrator/0.log"
Jul 09 18:29:44.148062 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:44.148033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-sgt5h_9bdd5402-1bab-497d-a86b-d5d5f1543a51/graceful-termination/0.log"
Jul 09 18:29:44.436735 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:44.436692    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-k2wql_7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949/kube-storage-version-migrator-operator/1.log"
Jul 09 18:29:44.438379 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:44.438335    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-k2wql_7da0fe58-3ce8-475a-9b6a-5cdf6f1d9949/kube-storage-version-migrator-operator/0.log"
Jul 09 18:29:45.433618 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.433588    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-25qdq_04cf20b8-27c4-48ec-bd17-69fa01f5256d/kube-multus/0.log"
Jul 09 18:29:45.476280 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.476251    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/kube-multus-additional-cni-plugins/0.log"
Jul 09 18:29:45.508094 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.508067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/egress-router-binary-copy/0.log"
Jul 09 18:29:45.539780 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.539756    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/cni-plugins/0.log"
Jul 09 18:29:45.576318 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.576292    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/bond-cni-plugin/0.log"
Jul 09 18:29:45.602538 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.602509    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/routeoverride-cni/0.log"
Jul 09 18:29:45.675737 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.675710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/whereabouts-cni-bincopy/0.log"
Jul 09 18:29:45.731586 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:45.731519    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-27sfj_69b094cf-1239-4c18-9511-db2b7c29c5b8/whereabouts-cni/0.log"
Jul 09 18:29:46.237662 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:46.237633    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbfs7_30a61f7d-2778-48d8-b750-6570eccbf1b5/network-metrics-daemon/0.log"
Jul 09 18:29:46.260662 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:46.260635    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbfs7_30a61f7d-2778-48d8-b750-6570eccbf1b5/kube-rbac-proxy/0.log"
Jul 09 18:29:47.513298 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.513265    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-controller/0.log"
Jul 09 18:29:47.531336 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.531302    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:29:47.531336 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.531326    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/1.log"
Jul 09 18:29:47.555074 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.555049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/kube-rbac-proxy-node/0.log"
Jul 09 18:29:47.585623 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.585601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 18:29:47.608356 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.608320    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/northd/0.log"
Jul 09 18:29:47.633289 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.633268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/nbdb/0.log"
Jul 09 18:29:47.658062 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.658036    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/sbdb/0.log"
Jul 09 18:29:47.727776 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:47.727741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovnkube-controller/0.log"
Jul 09 18:29:48.533319 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:48.533210    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:29:48.555053 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:48.544862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txgxp_c45a8f89-5a42-4d46-9d90-3f0718b5f72b/ovn-acl-logging/0.log"
Jul 09 18:29:48.790498 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:48.790407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-llc92_018d06aa-ce76-4485-95e0-e46b98c1f69c/network-check-target-container/0.log"
Jul 09 18:29:49.671780 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:49.671748    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5bbhv_54dc526f-a5fd-40fb-9e91-2f5784348b86/iptables-alerter/0.log"
Jul 09 18:29:50.383356 ip-10-0-136-100 kubenswrapper[2577]: I0709 18:29:50.383325    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9d5x8_105045a6-80b7-4c34-8cfd-03248d4e3b16/tuned/0.log"