Jul 28 16:01:14.839983 ip-10-0-142-124 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 16:01:15.198482 ip-10-0-142-124 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:15.198482 ip-10-0-142-124 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 16:01:15.198482 ip-10-0-142-124 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:15.198482 ip-10-0-142-124 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 16:01:15.198482 ip-10-0-142-124 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 16:01:15.200685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.200569    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.204970    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205000    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205004    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205008    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205011    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.205001 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205013    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205017    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205019    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205022    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205025    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205033    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205039    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205042    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205045    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205048    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205051    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205053    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205056    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205058    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205062    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205064    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205067    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205070    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205072    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205075    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.205176 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205078    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205080    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205083    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205085    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205087    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205090    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205093    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205095    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205098    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205102    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205106    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205108    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205111    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205113    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205116    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205120    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205123    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205126    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205129    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.205680 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205139    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205141    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205144    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205147    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205150    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205152    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205155    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205157    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205160    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205162    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205165    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205167    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205170    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205172    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205175    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205177    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205181    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205183    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205186    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205189    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.206185 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205191    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205193    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205196    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205198    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205201    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205203    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205206    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205210    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205213    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205215    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205217    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205220    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205223    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205225    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205228    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205232    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205234    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205236    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205239    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205242    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.206675 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205244    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205246    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205249    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205252    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205254    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205256    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205259    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205261    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205264    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205266    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205268    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205270    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205273    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205275    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.205279    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206014    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206022    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206026    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206029    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206032    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.207187 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206035    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206039    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206042    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206044    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206048    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206051    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206054    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206057    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206059    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206062    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206064    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206067    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206069    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206072    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206074    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206077    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206079    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206081    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206084    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206087    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.207691 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206089    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206092    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206094    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206097    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206100    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206102    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206105    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206108    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206111    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206113    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206117    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206119    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206122    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206125    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206127    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206130    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206132    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206136    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206138    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.208217 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206141    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206143    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206146    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206150    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206154    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206156    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206159    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206162    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206165    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206168    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206170    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206173    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206176    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206178    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206181    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206184    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206186    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206190    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206193    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.208701 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206196    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206199    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206202    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206206    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206208    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206212    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206214    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206217    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206220    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206222    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206225    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206228    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206231    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206235    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206238    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206241    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206244    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206247    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206250    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206252    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.209163 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206255    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206258    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206260    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206263    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206265    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206268    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206270    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206272    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206275    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206278    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206280    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206283    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206285    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206288    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206291    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206293    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206694    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206701    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206704    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206706    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.209664 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206709    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206712    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206714    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206717    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206720    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206725    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206728    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206732    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206734    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206737    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206740    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206744    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206747    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206749    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206752    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206755    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206757    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206760    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206762    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206765    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.210169 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206767    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206771    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206773    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206776    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206779    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206781    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206784    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206786    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206789    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206791    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206794    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206796    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206799    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206802    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206804    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206806    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206809    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206812    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206814    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206817    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.210669 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206819    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206824    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206827    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206830    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206833    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206835    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206838    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206840    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206842    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206845    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206847    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206850    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206852    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206855    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206858    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206860    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206863    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206866    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206868    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206871    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.211160 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206873    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206876    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206879    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206882    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206884    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206887    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206889    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206892    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206894    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206896    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206899    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206902    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206904    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206907    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206909    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206912    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206914    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206917    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206919    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206922    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.211742 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206924    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206926    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206929    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206932    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206934    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206937    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206941    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206944    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206947    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206950    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206953    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206956    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206958    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206961    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.206963    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207035    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207043    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207049    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207054    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207058    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207061    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 16:01:15.212226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207065    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207082    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207086    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207089    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207092    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207096    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207099    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207102    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207105    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207107    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207110    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207113    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207117    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207119    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207122    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207125    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207131    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207135    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207137    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207141    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207144    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207147    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207150    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207153    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 16:01:15.212784 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207156    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207159    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207163    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207166    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207169    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207171    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207174    2556 flags.go:64] FLAG: --enable-server="true"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207177    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207183    2556 flags.go:64] FLAG: --event-burst="100"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207186    2556 flags.go:64] FLAG: --event-qps="50"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207189    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207192    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207196    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207200    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207202    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207205    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207208    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207211    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207214    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207216    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207219    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207222    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207225    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207228    2556 flags.go:64] FLAG: --feature-gates=""
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207232    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 16:01:15.213362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207235    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207238    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207242    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207245    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207248    2556 flags.go:64] FLAG: --help="false"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207251    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207255    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207257    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207260    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207263    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207267    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207269    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207272    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207275    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207278    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207281    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207284    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207287    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207289    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207293    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207296    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207299    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207301    2556 flags.go:64] FLAG: --lock-file=""
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207304    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 16:01:15.214034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207307    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207310    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207315    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207318    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207320    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207323    2556 flags.go:64] FLAG: --logging-format="text"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207326    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207329    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207333    2556 flags.go:64] FLAG: --manifest-url=""
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207336    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207342    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207345    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207349    2556 flags.go:64] FLAG: --max-pods="110"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207353    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207356    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207358    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207361    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207364    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207367    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207370    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207377    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207380    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207383    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 16:01:15.214696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207386    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207389    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207395    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207398    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207401    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207404    2556 flags.go:64] FLAG: --port="10250"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207407    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207410    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-047668b32d6970a8a"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207413    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207416    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207418    2556 flags.go:64] FLAG: --register-node="true"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207421    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207425    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207428    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207430    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207433    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207437    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207440    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207443    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207446    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207450    2556 flags.go:64] FLAG: --runonce="false"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207453    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207456    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207459    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207462    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 16:01:15.215470 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207465    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207468    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207471    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207474    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207477    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207480    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207483    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207486    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207489    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207492    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207495    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207500    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207503    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207506    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207510    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207513    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207516    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207519    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207521    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207524    2556 flags.go:64] FLAG: --v="2"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207528    2556 flags.go:64] FLAG: --version="false"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207532    2556 flags.go:64] FLAG: --vmodule=""
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207536    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.207539    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207671    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.216172 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207675    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207678    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207684    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207688    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207691    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207694    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207697    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207700    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207703    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207705    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207708    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207711    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207714    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207717    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207719    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207722    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207725    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207727    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207729    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207732    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.216832 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207734    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207737    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207739    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207742    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207745    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207747    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207750    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207752    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207754    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207757    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207761    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207763    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207766    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207768    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207772    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207774    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207777    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207779    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207781    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.217378 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207785    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207788    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207791    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207795    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207798    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207800    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207803    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207806    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207809    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207811    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207814    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207817    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207819    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207822    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207824    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207829    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207832    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207835    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207838    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.217877 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207840    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207843    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207845    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207848    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207852    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207854    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207857    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207859    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207863    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207865    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207868    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207870    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207873    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207875    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207878    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207881    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207883    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207886    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207889    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207891    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.218327 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207894    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207897    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207899    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207901    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207904    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207906    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207908    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207911    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207913    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207916    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207919    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207921    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207923    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207926    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207928    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207931    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207934    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207937    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207939    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.218839 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.207942    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.208571    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.217698    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.217722    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.217744    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.217754    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217814    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217819    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217822    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217825    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217828    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217830    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217833    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217836    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217839    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.219297 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217841    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217844    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217847    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217851    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217855    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217858    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217861    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217863    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217866    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217869    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217872    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217875    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217877    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217880    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217886    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217891    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217894    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217896    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217899    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.219694 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217902    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217906    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217908    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217912    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217914    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217917    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217919    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217922    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217924    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217927    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217929    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217932    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217934    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217936    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217939    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217941    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217943    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217946    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217948    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.220188 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217951    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217953    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217956    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217959    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217961    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217964    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217966    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217969    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217972    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217974    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217977    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217979    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217982    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217985    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217988    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217992    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217995    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.217998    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218000    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.220670 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218003    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218005    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218008    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218010    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218012    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218015    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218018    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218020    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218022    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218025    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218027    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218030    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218032    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218035    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218037    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218040    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218042    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218045    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218047    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218049    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.221135 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218052    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218054    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218057    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218060    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218062    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218065    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218067    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218070    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218072    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218075    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218077    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218080    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218082    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.218087    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218207    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 16:01:15.221649 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218212    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218215    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218219    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218221    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218225    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218228    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218230    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218233    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218236    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218238    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218241    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218245    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218247    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218250    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218252    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218255    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218257    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218260    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218262    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 16:01:15.222026 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218265    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218268    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218271    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218273    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218276    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218279    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218281    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218283    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218286    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218289    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218291    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218293    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218296    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218298    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218301    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218303    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218305    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218308    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218311    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218314    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 16:01:15.222481 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218316    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218319    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218321    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218324    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218327    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218330    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218335    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218338    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218341    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218344    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218346    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218350    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218352    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218355    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218358    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218360    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218363    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218365    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218367    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 16:01:15.222988 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218370    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218372    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218374    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218377    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218379    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218382    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218384    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218386    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218389    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218391    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218393    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218396    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218398    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218400    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218403    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218405    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218410    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218412    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218415    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218418    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 16:01:15.223446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218421    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218423    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218425    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218428    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218430    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218433    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218436    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218438    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218441    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218443    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218446    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218449    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218451    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218453    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218457    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218460    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218463    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218466    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218469    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 16:01:15.223957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:15.218471    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.218476    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.218605    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.220939    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.222327    2556 server.go:1019] "Starting client certificate rotation"
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.222430    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:01:15.224414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.222468    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 16:01:15.224721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.224705    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:01:15.225990 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.225962    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 16:01:15.236440 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.236421    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 28 16:01:15.239554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.239535    2556 log.go:25] "Validated CRI v1 image API"
Jul 28 16:01:15.240411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.240394    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 16:01:15.242615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.242591    2556 fs.go:135] Filesystem UUIDs: map[00e268e2-0c11-49cd-b07e-5f1c50662417:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 99e40ba2-ff0a-4156-8807-30bc41133f92:/dev/nvme0n1p3]
Jul 28 16:01:15.242691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.242615    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 16:01:15.249742 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.249591    2556 manager.go:216] Machine: {Timestamp:2026-07-28 16:01:15.248285829 +0000 UTC m=+0.299089892 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100512 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2604795d5602e1c8f7ccc3478b947e SystemUUID:ec260479-5d56-02e1-c8f7-ccc3478b947e BootID:9685734c-86ed-4215-89d4-0c0782b4dd97 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2f:8a:cd:8d:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2f:8a:cd:8d:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:26:4a:22:8a:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 16:01:15.249742 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.249727    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 16:01:15.249893 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.249825    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 16:01:15.251666 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.251647    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:01:15.252909 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.252882    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 16:01:15.253064 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.252912    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 16:01:15.253653 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.253643    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 16:01:15.253690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.253656    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 16:01:15.253690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.253664    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:01:15.253690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.253678    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 16:01:15.254822 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.254812    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 16:01:15.256532 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256522    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:01:15.256665 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256656    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 16:01:15.256696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256686    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 16:01:15.256696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256696    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 16:01:15.256754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256706    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 16:01:15.256754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256715    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 28 16:01:15.256754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.256723    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 16:01:15.257202 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.257182    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:01:15.257202 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.257205    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 16:01:15.259116 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.259095    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 16:01:15.261279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.261265    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 16:01:15.261330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.261296    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 16:01:15.263042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263017    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 16:01:15.263042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263036    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 16:01:15.263042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263042    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 16:01:15.263042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263048    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263059    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263067    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263076    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263082    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263090    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263095    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263104    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 16:01:15.263260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.263110    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 16:01:15.264560 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.264546    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 16:01:15.264606 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.264565    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 16:01:15.267837 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.267798    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 16:01:15.267837 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.267813    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 16:01:15.267980 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.267921    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 16:01:15.268437 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.268420    2556 server.go:1268] "Started kubelet"
Jul 28 16:01:15.268524 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.268487    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 16:01:15.269242 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.269157    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 16:01:15.269336 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.269263    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 16:01:15.269582 ip-10-0-142-124 systemd[1]: Started Kubernetes Kubelet.
Jul 28 16:01:15.271520 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.271499    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 16:01:15.271610 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.271551    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 16:01:15.276178 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.276150    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 16:01:15.276962 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.276072    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-124.ec2.internal.18c67f7c900c7624  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-124.ec2.internal,UID:ip-10-0-142-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-124.ec2.internal,},FirstTimestamp:2026-07-28 16:01:15.268388388 +0000 UTC m=+0.319192453,LastTimestamp:2026-07-28 16:01:15.268388388 +0000 UTC m=+0.319192453,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-124.ec2.internal,}"
Jul 28 16:01:15.278539 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.278521    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vr5n4"
Jul 28 16:01:15.279260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279247    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 16:01:15.279314 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279248    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 16:01:15.279423 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.279404    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.279846 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279824    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 16:01:15.279846 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279824    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 16:01:15.280020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279855    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 16:01:15.280020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279967    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 16:01:15.280020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.279977    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 16:01:15.280268 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280250    2556 factory.go:55] Registering systemd factory
Jul 28 16:01:15.280268 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280270    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 28 16:01:15.280467 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280453    2556 factory.go:153] Registering CRI-O factory
Jul 28 16:01:15.280528 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280482    2556 factory.go:223] Registration of the crio container factory successfully
Jul 28 16:01:15.280582 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280540    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 16:01:15.280582 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280566    2556 factory.go:103] Registering Raw factory
Jul 28 16:01:15.280695 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280584    2556 manager.go:1196] Started watching for new ooms in manager
Jul 28 16:01:15.280974 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.280961    2556 manager.go:349] Starting recovery of all containers
Jul 28 16:01:15.281586 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.281561    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 16:01:15.281803 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.281781    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 16:01:15.288583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.288453    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vr5n4"
Jul 28 16:01:15.291237 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.291221    2556 manager.go:354] Recovery completed
Jul 28 16:01:15.295932 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.295918    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.297919 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.297900    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.298005 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.297931    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.298005 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.297941    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.298527 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.298512    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 16:01:15.298527 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.298524    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 16:01:15.298686 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.298534    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.298686 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.298544    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 16:01:15.299254 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.299238    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.299324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.299267    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.299324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.299278    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.300014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.300000    2556 policy_none.go:49] "None policy: Start"
Jul 28 16:01:15.300075 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.300021    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 16:01:15.300075 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.300045    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 16:01:15.301023 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.301009    2556 policy_none.go:47] "Start"
Jul 28 16:01:15.340396 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340363    2556 manager.go:337] "Starting Device Plugin manager"
Jul 28 16:01:15.340530 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.340410    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 16:01:15.340530 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340423    2556 server.go:87] "Starting device plugin registration server"
Jul 28 16:01:15.340789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340774    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 16:01:15.340906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340790    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 16:01:15.340906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340868    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 16:01:15.341005 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340954    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 16:01:15.341005 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.340960    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 16:01:15.341559 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.341534    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 16:01:15.341679 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.341597    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.441935 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.441897    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.443356 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.443339    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.443434 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.443376    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.443434 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.443391    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.443434 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.443418    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.452581 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.452519    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.452581 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.452544    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-124.ec2.internal\": node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.454725 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.454703    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 16:01:15.456172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.456149    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 16:01:15.456270 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.456188    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 16:01:15.456270 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.456211    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 16:01:15.456270 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.456246    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 16:01:15.460317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.460290    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:15.556755 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.556713    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"]
Jul 28 16:01:15.556897 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.556806    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.558655 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.558640    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.558734 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.558670    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.558734 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.558680    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.559910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.559897    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.560020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560008    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.560053 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560037    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.560722 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560707    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.560808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560738    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.560808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560750    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.560808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560751    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.560808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560774    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.560808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.560785    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.561849 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.561834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.561900 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.561868    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 16:01:15.562598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.562583    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 16:01:15.562693 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.562609    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 16:01:15.562693 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.562643    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeHasSufficientPID"
Jul 28 16:01:15.562974 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.562960    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.580713 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.580685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/834e29422605a4563abd0c2b2da8dece-config\") pod \"kube-apiserver-proxy-ip-10-0-142-124.ec2.internal\" (UID: \"834e29422605a4563abd0c2b2da8dece\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.580825 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.580717    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.580825 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.580735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.589512 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.589490    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-124.ec2.internal\" not found" node="ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.594313 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.594295    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-124.ec2.internal\" not found" node="ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.664055 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.664013    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.681415 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/834e29422605a4563abd0c2b2da8dece-config\") pod \"kube-apiserver-proxy-ip-10-0-142-124.ec2.internal\" (UID: \"834e29422605a4563abd0c2b2da8dece\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.681415 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.681612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.681612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/834e29422605a4563abd0c2b2da8dece-config\") pod \"kube-apiserver-proxy-ip-10-0-142-124.ec2.internal\" (UID: \"834e29422605a4563abd0c2b2da8dece\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.681612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.681612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.681577    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35429a64cc41bf0a3ba185325ae3dc4f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal\" (UID: \"35429a64cc41bf0a3ba185325ae3dc4f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.764308 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.764233    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.865053 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.865021    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:15.892539 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.892505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.898701 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:15.898682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:15.965480 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:15.965423    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:16.066139 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.066063    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-124.ec2.internal\" not found"
Jul 28 16:01:16.098294 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.098269    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:16.164891 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.164859    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:16.179502 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.179461    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:16.197740 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.197715    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:01:16.199489 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.199472    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal"
Jul 28 16:01:16.213750 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.213726    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 16:01:16.221371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.221343    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 16:01:16.221598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.221566    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:01:16.221598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.221589    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:01:16.221775 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.221578    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 16:01:16.257866 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.257811    2556 apiserver.go:52] "Watching apiserver"
Jul 28 16:01:16.278184 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.278155    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 16:01:16.279383 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.279358    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7","openshift-cluster-node-tuning-operator/tuned-45wh8","openshift-dns/node-resolver-6j4ql","openshift-image-registry/node-ca-69jt2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-w544q","openshift-multus/multus-fshgz","kube-system/global-pull-secret-syncer-j225x","kube-system/konnectivity-agent-rcrxp","openshift-multus/network-metrics-daemon-v2hxs","openshift-network-diagnostics/network-check-target-bfrcj","openshift-network-operator/iptables-alerter-r9jkf","openshift-ovn-kubernetes/ovnkube-node-mlq9q"]
Jul 28 16:01:16.279478 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.279398    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 16:01:16.281046 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.281024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.282196 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.282177    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.284871 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.284832    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.284894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-registration-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.284927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-device-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.284952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.284977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-socket-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-sys-fs\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285030    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78mdl\" (UniqueName: \"kubernetes.io/projected/3ea2d5c9-69d3-45e2-b740-888c50866a86-kube-api-access-78mdl\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285190    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285199    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ms8sd\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285196    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285302    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285329    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285330    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.285393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285333    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hw442\""
Jul 28 16:01:16.285670 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.285412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.286479 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.286464    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.288167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.288149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.288931 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.288910    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-89p94\""
Jul 28 16:01:16.289043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.288966    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.289315 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.289297    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.289397 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.289383    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 16:01:16.289583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.289567    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2m5v7\""
Jul 28 16:01:16.289730 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.289714    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.289803 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.289789    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.290426 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.290408    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 16:01:16.290508 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.290466    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.290508 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.290483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.290786 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.290771    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.290884 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.290844    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:16.291161 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.291146    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 16:01:16.292039 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.292022    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.292373 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.292344    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jgl8b\""
Jul 28 16:01:16.292482 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.292346    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 16:01:16.292482 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.292402    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.293119 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.293103    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.293205 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.293183    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:16.294117 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.294082    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 15:56:15 +0000 UTC" deadline="2028-03-18 04:14:58.594594583 +0000 UTC"
Jul 28 16:01:16.294184 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.294118    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14364h13m42.300479323s"
Jul 28 16:01:16.294241 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.294201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:16.294292 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.294268    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:16.294754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.294681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 16:01:16.294754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.294732    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l9sfg\""
Jul 28 16:01:16.295393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.295380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.296521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.296502    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 16:01:16.296783 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.296767    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t9nkh\""
Jul 28 16:01:16.296783 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.296777    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 16:01:16.297072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.297055    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.298674 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.298655    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.298807 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.298692    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-smpt7\""
Jul 28 16:01:16.298807 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.298654    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 16:01:16.298807 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.298739    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.300792 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.300319    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 16:01:16.301573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301405    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 16:01:16.301573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301427    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 16:01:16.301573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301427    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 16:01:16.301573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301501    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 16:01:16.301573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301516    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 16:01:16.301867 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301719    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jsdfc\""
Jul 28 16:01:16.301867 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.301755    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 16:01:16.337365 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.337339    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvqwk"
Jul 28 16:01:16.346329 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.346297    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvqwk"
Jul 28 16:01:16.380930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.380906    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 16:01:16.385191 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-socket-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.385317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-cnibin\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.385317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:16.385317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385262    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.385317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/811710f4-2c68-43dd-a1df-f3cad45012c1-agent-certs\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.385317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w52n\" (UniqueName: \"kubernetes.io/projected/9da7f48a-38cf-4878-bb8d-53c58670b5d7-kube-api-access-7w52n\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-etc-kubernetes\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-tuned\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvpwh\" (UniqueName: \"kubernetes.io/projected/28ef4649-bfed-43a0-aabc-301174665b34-kube-api-access-lvpwh\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-socket-dir-parent\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385437    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6c5\" (UniqueName: \"kubernetes.io/projected/df7f31c1-0c68-4404-bdf1-e4be1580385f-kube-api-access-4n6c5\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425eac07-3ae4-4c5a-94b6-839025d8d7ed-host\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-dbus\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385540    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-script-lib\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.385578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-cnibin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-cni-binary-copy\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-tmp\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385660    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj29q\" (UniqueName: \"kubernetes.io/projected/53fa9439-4cdf-4b78-8e90-7bc32b242d54-kube-api-access-lj29q\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385712    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-hostroot\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-systemd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-var-lib-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385828    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-bin\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-k8s-cni-cncf-io\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-log-socket\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-bin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-daemon-config\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.385966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-conf\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.386025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-slash\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edd4ba9f-f16f-42de-ac26-975c51735d1a-hosts-file\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwsn\" (UniqueName: \"kubernetes.io/projected/edd4ba9f-f16f-42de-ac26-975c51735d1a-kube-api-access-ptwsn\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-conf-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-multus-certs\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386189    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysconfig\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78mdl\" (UniqueName: \"kubernetes.io/projected/3ea2d5c9-69d3-45e2-b740-888c50866a86-kube-api-access-78mdl\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-os-release\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386258    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-modprobe-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-kubernetes\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386298    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-run\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-device-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-netns\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-ovn\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386516    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-device-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.386615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386557    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-multus\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386587    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-kubelet\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-systemd\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-socket-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdq6l\" (UniqueName: \"kubernetes.io/projected/425eac07-3ae4-4c5a-94b6-839025d8d7ed-kube-api-access-fdq6l\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386741    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-node-log\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386769    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d54v9\" (UniqueName: \"kubernetes.io/projected/7901a8b4-34c9-4668-b0f9-759fff4e035f-kube-api-access-d54v9\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd4ba9f-f16f-42de-ac26-975c51735d1a-tmp-dir\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386814    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-os-release\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/811710f4-2c68-43dd-a1df-f3cad45012c1-konnectivity-ca\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-config\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.386977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-system-cni-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.387324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425eac07-3ae4-4c5a-94b6-839025d8d7ed-serviceca\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-kubelet-config\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-sys-fs\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387131    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-netns\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-host\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-registration-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387185    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-sys-fs\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-kubelet\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-netd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e9014d-d847-44b4-91a3-b6c78730faed-iptables-alerter-script\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387242    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-registration-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e9014d-d847-44b4-91a3-b6c78730faed-host-slash\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvzk\" (UniqueName: \"kubernetes.io/projected/a3e9014d-d847-44b4-91a3-b6c78730faed-kube-api-access-pxvzk\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-var-lib-kubelet\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-etc-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovn-node-metrics-cert\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.387969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-systemd-units\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387476    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea2d5c9-69d3-45e2-b740-888c50866a86-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-env-overrides\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387546    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-binary-copy\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-system-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-sys\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.388442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.387662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-lib-modules\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.401609 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.401582    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 16:01:16.405065 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.405036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78mdl\" (UniqueName: \"kubernetes.io/projected/3ea2d5c9-69d3-45e2-b740-888c50866a86-kube-api-access-78mdl\") pod \"aws-ebs-csi-driver-node-kbfm7\" (UID: \"3ea2d5c9-69d3-45e2-b740-888c50866a86\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.488816 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488781    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwsn\" (UniqueName: \"kubernetes.io/projected/edd4ba9f-f16f-42de-ac26-975c51735d1a-kube-api-access-ptwsn\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.488816 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-conf-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-multus-certs\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysconfig\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-multus-certs\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488904    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-conf-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488920    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-os-release\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488955    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-modprobe-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-os-release\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-kubernetes\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.488998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysconfig\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-kubernetes\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489027    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-run\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-netns\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-ovn\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489100    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-modprobe-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-multus\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489121    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-run\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-netns\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-multus\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-ovn\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-kubelet\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-kubelet\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489189    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-systemd\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdq6l\" (UniqueName: \"kubernetes.io/projected/425eac07-3ae4-4c5a-94b6-839025d8d7ed-kube-api-access-fdq6l\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-node-log\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d54v9\" (UniqueName: \"kubernetes.io/projected/7901a8b4-34c9-4668-b0f9-759fff4e035f-kube-api-access-d54v9\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489263    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-systemd\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd4ba9f-f16f-42de-ac26-975c51735d1a-tmp-dir\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489278    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-node-log\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.489790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-os-release\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/811710f4-2c68-43dd-a1df-f3cad45012c1-konnectivity-ca\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489371    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-config\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-os-release\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-system-cni-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425eac07-3ae4-4c5a-94b6-839025d8d7ed-serviceca\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-system-cni-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-kubelet-config\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-netns\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-host\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-kubelet\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489554    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-kubelet-config\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-netd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-netns\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489617    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e9014d-d847-44b4-91a3-b6c78730faed-iptables-alerter-script\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.490753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489674    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-kubelet\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489702    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e9014d-d847-44b4-91a3-b6c78730faed-host-slash\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-host\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edd4ba9f-f16f-42de-ac26-975c51735d1a-tmp-dir\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489734    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvzk\" (UniqueName: \"kubernetes.io/projected/a3e9014d-d847-44b4-91a3-b6c78730faed-kube-api-access-pxvzk\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-netd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-var-lib-kubelet\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489807    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3e9014d-d847-44b4-91a3-b6c78730faed-host-slash\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-etc-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-var-lib-kubelet\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489915    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovn-node-metrics-cert\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489949    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-etc-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.489993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/425eac07-3ae4-4c5a-94b6-839025d8d7ed-serviceca\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-systemd-units\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/811710f4-2c68-43dd-a1df-f3cad45012c1-konnectivity-ca\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.491824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-env-overrides\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-binary-copy\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-system-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-sys\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-config\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490158    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-lib-modules\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490185    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-systemd-units\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-cnibin\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-system-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a3e9014d-d847-44b4-91a3-b6c78730faed-iptables-alerter-script\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-sys\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490332    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/811710f4-2c68-43dd-a1df-f3cad45012c1-agent-certs\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w52n\" (UniqueName: \"kubernetes.io/projected/9da7f48a-38cf-4878-bb8d-53c58670b5d7-kube-api-access-7w52n\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-etc-kubernetes\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.492680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-tuned\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvpwh\" (UniqueName: \"kubernetes.io/projected/28ef4649-bfed-43a0-aabc-301174665b34-kube-api-access-lvpwh\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-etc-kubernetes\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-socket-dir-parent\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-env-overrides\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-d\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490567    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6c5\" (UniqueName: \"kubernetes.io/projected/df7f31c1-0c68-4404-bdf1-e4be1580385f-kube-api-access-4n6c5\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490586    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425eac07-3ae4-4c5a-94b6-839025d8d7ed-host\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490616    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-dbus\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-script-lib\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490654    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-socket-dir-parent\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-cni-binary-copy\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-cnibin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28ef4649-bfed-43a0-aabc-301174665b34-cnibin\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490746    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/425eac07-3ae4-4c5a-94b6-839025d8d7ed-host\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-lib-modules\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-cni-binary-copy\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.493563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490909    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-tmp\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj29q\" (UniqueName: \"kubernetes.io/projected/53fa9439-4cdf-4b78-8e90-7bc32b242d54-kube-api-access-lj29q\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-hostroot\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.490990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-systemd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-var-lib-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-dbus\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-bin\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-cni-bin\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-cnibin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491172    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-hostroot\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovnkube-script-lib\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491302    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-run-systemd\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-var-lib-openvswitch\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491574    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-k8s-cni-cncf-io\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491625    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/28ef4649-bfed-43a0-aabc-301174665b34-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491643    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-log-socket\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-bin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-cni-dir\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-var-lib-cni-bin\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/df7f31c1-0c68-4404-bdf1-e4be1580385f-host-run-k8s-cni-cncf-io\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-log-socket\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-daemon-config\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-cni-binary-copy\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-conf\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.491966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-slash\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-sysctl-conf\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492063    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edd4ba9f-f16f-42de-ac26-975c51735d1a-hosts-file\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.492107    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/edd4ba9f-f16f-42de-ac26-975c51735d1a-hosts-file\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.492201    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:16.992164576 +0000 UTC m=+2.042968629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:16.494765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-slash\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492253    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/df7f31c1-0c68-4404-bdf1-e4be1580385f-multus-daemon-config\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.492258    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7901a8b4-34c9-4668-b0f9-759fff4e035f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.492265    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.492314    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:16.992301579 +0000 UTC m=+2.043105633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.493397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-etc-tuned\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.493495    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7901a8b4-34c9-4668-b0f9-759fff4e035f-ovn-node-metrics-cert\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.493571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53fa9439-4cdf-4b78-8e90-7bc32b242d54-tmp\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.495275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.493651    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/811710f4-2c68-43dd-a1df-f3cad45012c1-agent-certs\") pod \"konnectivity-agent-rcrxp\" (UID: \"811710f4-2c68-43dd-a1df-f3cad45012c1\") " pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.499045 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.498828    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:16.499045 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.498856    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:16.499045 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.498872    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:16.499045 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.498944    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:16.998925095 +0000 UTC m=+2.049729156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:16.499045 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.499012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvzk\" (UniqueName: \"kubernetes.io/projected/a3e9014d-d847-44b4-91a3-b6c78730faed-kube-api-access-pxvzk\") pod \"iptables-alerter-r9jkf\" (UID: \"a3e9014d-d847-44b4-91a3-b6c78730faed\") " pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.499394 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.499123    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdq6l\" (UniqueName: \"kubernetes.io/projected/425eac07-3ae4-4c5a-94b6-839025d8d7ed-kube-api-access-fdq6l\") pod \"node-ca-69jt2\" (UID: \"425eac07-3ae4-4c5a-94b6-839025d8d7ed\") " pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.499568 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.499507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d54v9\" (UniqueName: \"kubernetes.io/projected/7901a8b4-34c9-4668-b0f9-759fff4e035f-kube-api-access-d54v9\") pod \"ovnkube-node-mlq9q\" (UID: \"7901a8b4-34c9-4668-b0f9-759fff4e035f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.499909 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.499890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwsn\" (UniqueName: \"kubernetes.io/projected/edd4ba9f-f16f-42de-ac26-975c51735d1a-kube-api-access-ptwsn\") pod \"node-resolver-6j4ql\" (UID: \"edd4ba9f-f16f-42de-ac26-975c51735d1a\") " pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.500385 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.500362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6c5\" (UniqueName: \"kubernetes.io/projected/df7f31c1-0c68-4404-bdf1-e4be1580385f-kube-api-access-4n6c5\") pod \"multus-fshgz\" (UID: \"df7f31c1-0c68-4404-bdf1-e4be1580385f\") " pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.501449 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.501415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvpwh\" (UniqueName: \"kubernetes.io/projected/28ef4649-bfed-43a0-aabc-301174665b34-kube-api-access-lvpwh\") pod \"multus-additional-cni-plugins-w544q\" (UID: \"28ef4649-bfed-43a0-aabc-301174665b34\") " pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.501525 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.501498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w52n\" (UniqueName: \"kubernetes.io/projected/9da7f48a-38cf-4878-bb8d-53c58670b5d7-kube-api-access-7w52n\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.503084 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.503065    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj29q\" (UniqueName: \"kubernetes.io/projected/53fa9439-4cdf-4b78-8e90-7bc32b242d54-kube-api-access-lj29q\") pod \"tuned-45wh8\" (UID: \"53fa9439-4cdf-4b78-8e90-7bc32b242d54\") " pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.606176 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.606129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7"
Jul 28 16:01:16.639374 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.639336    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-45wh8"
Jul 28 16:01:16.642931 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.642909    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:16.658168 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.658120    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6j4ql"
Jul 28 16:01:16.670319 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.670278    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35429a64cc41bf0a3ba185325ae3dc4f.slice/crio-a2c5cee4b6aa6c4671c83888e1c26f330ccb8cbf2c4b70085b3ba763eda83768 WatchSource:0}: Error finding container a2c5cee4b6aa6c4671c83888e1c26f330ccb8cbf2c4b70085b3ba763eda83768: Status 404 returned error can't find the container with id a2c5cee4b6aa6c4671c83888e1c26f330ccb8cbf2c4b70085b3ba763eda83768
Jul 28 16:01:16.670852 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.670829    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fa9439_4cdf_4b78_8e90_7bc32b242d54.slice/crio-306f11c84be59bd1d24ec4fb34327b8c0498a005fb4a3ab83cf4ebd1e23dd83e WatchSource:0}: Error finding container 306f11c84be59bd1d24ec4fb34327b8c0498a005fb4a3ab83cf4ebd1e23dd83e: Status 404 returned error can't find the container with id 306f11c84be59bd1d24ec4fb34327b8c0498a005fb4a3ab83cf4ebd1e23dd83e
Jul 28 16:01:16.671229 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.671206    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd4ba9f_f16f_42de_ac26_975c51735d1a.slice/crio-462b2a4e8ae041192f2dd239c3c08b42d55f4960b8e9606aee7fb42f9df60503 WatchSource:0}: Error finding container 462b2a4e8ae041192f2dd239c3c08b42d55f4960b8e9606aee7fb42f9df60503: Status 404 returned error can't find the container with id 462b2a4e8ae041192f2dd239c3c08b42d55f4960b8e9606aee7fb42f9df60503
Jul 28 16:01:16.673488 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.673465    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69jt2"
Jul 28 16:01:16.676019 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.676001    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:01:16.682892 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.682865    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425eac07_3ae4_4c5a_94b6_839025d8d7ed.slice/crio-59c8fd351ae475f2a059baf6b9e5c48b568ef34005c4b1b920ca3062af79eb5f WatchSource:0}: Error finding container 59c8fd351ae475f2a059baf6b9e5c48b568ef34005c4b1b920ca3062af79eb5f: Status 404 returned error can't find the container with id 59c8fd351ae475f2a059baf6b9e5c48b568ef34005c4b1b920ca3062af79eb5f
Jul 28 16:01:16.690715 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.690691    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w544q"
Jul 28 16:01:16.694890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.694868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fshgz"
Jul 28 16:01:16.699041 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.699004    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ef4649_bfed_43a0_aabc_301174665b34.slice/crio-dc0e534f4693b419512415bbf6814fba56300b3c6328f19326d86bab10aeb344 WatchSource:0}: Error finding container dc0e534f4693b419512415bbf6814fba56300b3c6328f19326d86bab10aeb344: Status 404 returned error can't find the container with id dc0e534f4693b419512415bbf6814fba56300b3c6328f19326d86bab10aeb344
Jul 28 16:01:16.703554 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.703531    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7f31c1_0c68_4404_bdf1_e4be1580385f.slice/crio-ec10638dead8f47d6fedb41b2623c152ba433ef19677c6fd855e46128456277c WatchSource:0}: Error finding container ec10638dead8f47d6fedb41b2623c152ba433ef19677c6fd855e46128456277c: Status 404 returned error can't find the container with id ec10638dead8f47d6fedb41b2623c152ba433ef19677c6fd855e46128456277c
Jul 28 16:01:16.715924 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.715895    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:16.722377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.722347    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r9jkf"
Jul 28 16:01:16.725976 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.725949    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811710f4_2c68_43dd_a1df_f3cad45012c1.slice/crio-9ff428e91c4ddbe96c08ddfe391bac7a730a7fa2df6fea09dc88e30f8f35a203 WatchSource:0}: Error finding container 9ff428e91c4ddbe96c08ddfe391bac7a730a7fa2df6fea09dc88e30f8f35a203: Status 404 returned error can't find the container with id 9ff428e91c4ddbe96c08ddfe391bac7a730a7fa2df6fea09dc88e30f8f35a203
Jul 28 16:01:16.727183 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.727154    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:16.730645 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.730615    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e9014d_d847_44b4_91a3_b6c78730faed.slice/crio-345bab9e8851c516c3d474fb6043346998520f24012b71bbd00182a2db891bd5 WatchSource:0}: Error finding container 345bab9e8851c516c3d474fb6043346998520f24012b71bbd00182a2db891bd5: Status 404 returned error can't find the container with id 345bab9e8851c516c3d474fb6043346998520f24012b71bbd00182a2db891bd5
Jul 28 16:01:16.736066 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.736037    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7901a8b4_34c9_4668_b0f9_759fff4e035f.slice/crio-78ddb88f1382e2181d168b381abd68f26f3dd6bc5d7bfcbce670be760e7b33a6 WatchSource:0}: Error finding container 78ddb88f1382e2181d168b381abd68f26f3dd6bc5d7bfcbce670be760e7b33a6: Status 404 returned error can't find the container with id 78ddb88f1382e2181d168b381abd68f26f3dd6bc5d7bfcbce670be760e7b33a6
Jul 28 16:01:16.969815 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:16.969778    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea2d5c9_69d3_45e2_b740_888c50866a86.slice/crio-0780cf2847ab4132e6ed9b9685257db4f35ca1e1b73cdac8fd6dfb7b85caf4d7 WatchSource:0}: Error finding container 0780cf2847ab4132e6ed9b9685257db4f35ca1e1b73cdac8fd6dfb7b85caf4d7: Status 404 returned error can't find the container with id 0780cf2847ab4132e6ed9b9685257db4f35ca1e1b73cdac8fd6dfb7b85caf4d7
Jul 28 16:01:16.996422 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.996381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:16.996422 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:16.996424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:16.996677 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.996544    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:16.996677 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.996545    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:16.996677 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.996604    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:17.996588411 +0000 UTC m=+3.047392463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:16.996677 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:16.996618    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:17.99661156 +0000 UTC m=+3.047415612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:17.097750 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.097713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:17.097949 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:17.097927    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:17.098026 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:17.097958    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:17.098026 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:17.097972    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:17.098130 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:17.098039    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:18.098023948 +0000 UTC m=+3.148828015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:17.258280 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.258196    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:17.348739 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.348641    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:56:16 +0000 UTC" deadline="2028-01-20 22:34:31.06967007 +0000 UTC"
Jul 28 16:01:17.348739 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.348731    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12990h33m13.720943213s"
Jul 28 16:01:17.457868 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.457834    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:17.458066 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:17.457975    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:17.462791 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.462732    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-45wh8" event={"ID":"53fa9439-4cdf-4b78-8e90-7bc32b242d54","Type":"ContainerStarted","Data":"306f11c84be59bd1d24ec4fb34327b8c0498a005fb4a3ab83cf4ebd1e23dd83e"}
Jul 28 16:01:17.465825 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.465781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"78ddb88f1382e2181d168b381abd68f26f3dd6bc5d7bfcbce670be760e7b33a6"}
Jul 28 16:01:17.467222 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.467189    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fshgz" event={"ID":"df7f31c1-0c68-4404-bdf1-e4be1580385f","Type":"ContainerStarted","Data":"ec10638dead8f47d6fedb41b2623c152ba433ef19677c6fd855e46128456277c"}
Jul 28 16:01:17.469841 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.469807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69jt2" event={"ID":"425eac07-3ae4-4c5a-94b6-839025d8d7ed","Type":"ContainerStarted","Data":"59c8fd351ae475f2a059baf6b9e5c48b568ef34005c4b1b920ca3062af79eb5f"}
Jul 28 16:01:17.472261 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.472224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7" event={"ID":"3ea2d5c9-69d3-45e2-b740-888c50866a86","Type":"ContainerStarted","Data":"0780cf2847ab4132e6ed9b9685257db4f35ca1e1b73cdac8fd6dfb7b85caf4d7"}
Jul 28 16:01:17.475103 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.475073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9jkf" event={"ID":"a3e9014d-d847-44b4-91a3-b6c78730faed","Type":"ContainerStarted","Data":"345bab9e8851c516c3d474fb6043346998520f24012b71bbd00182a2db891bd5"}
Jul 28 16:01:17.478702 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.478649    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcrxp" event={"ID":"811710f4-2c68-43dd-a1df-f3cad45012c1","Type":"ContainerStarted","Data":"9ff428e91c4ddbe96c08ddfe391bac7a730a7fa2df6fea09dc88e30f8f35a203"}
Jul 28 16:01:17.480774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.480741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerStarted","Data":"dc0e534f4693b419512415bbf6814fba56300b3c6328f19326d86bab10aeb344"}
Jul 28 16:01:17.483643 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.483600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j4ql" event={"ID":"edd4ba9f-f16f-42de-ac26-975c51735d1a","Type":"ContainerStarted","Data":"462b2a4e8ae041192f2dd239c3c08b42d55f4960b8e9606aee7fb42f9df60503"}
Jul 28 16:01:17.485185 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:17.485156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal" event={"ID":"35429a64cc41bf0a3ba185325ae3dc4f","Type":"ContainerStarted","Data":"a2c5cee4b6aa6c4671c83888e1c26f330ccb8cbf2c4b70085b3ba763eda83768"}
Jul 28 16:01:18.005804 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.005610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:18.005804 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.005686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:18.005804 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.005781    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:18.006148 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.005827    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:18.006148 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.005887    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:20.005844017 +0000 UTC m=+5.056648087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:18.006148 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.005910    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:20.005898275 +0000 UTC m=+5.056702331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:18.106359 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.106318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:18.106573 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.106502    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:18.106573 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.106528    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:18.106573 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.106541    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:18.106782 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.106614    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:20.106594122 +0000 UTC m=+5.157398177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:18.277782 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.277744    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:18.284382 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:01:18.284334    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834e29422605a4563abd0c2b2da8dece.slice/crio-d3c5b3baef4000a0ab9f3beefed3d2131e242758bc4aed874c1efbd645b06805 WatchSource:0}: Error finding container d3c5b3baef4000a0ab9f3beefed3d2131e242758bc4aed874c1efbd645b06805: Status 404 returned error can't find the container with id d3c5b3baef4000a0ab9f3beefed3d2131e242758bc4aed874c1efbd645b06805
Jul 28 16:01:18.351741 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.349253    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 15:56:16 +0000 UTC" deadline="2028-02-21 16:28:13.265891514 +0000 UTC"
Jul 28 16:01:18.351741 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.349292    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13752h26m54.91660459s"
Jul 28 16:01:18.457041 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.456979    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:18.457291 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.457124    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:18.457588 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.457543    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:18.457716 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:18.457666    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:18.522217 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:18.522006    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal" event={"ID":"834e29422605a4563abd0c2b2da8dece","Type":"ContainerStarted","Data":"d3c5b3baef4000a0ab9f3beefed3d2131e242758bc4aed874c1efbd645b06805"}
Jul 28 16:01:19.101074 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:19.101035    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 16:01:19.472828 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:19.472175    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:19.472828 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:19.472342    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:20.023904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:20.023958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.024107    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.024172    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:24.024151211 +0000 UTC m=+9.074955269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.024564    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:20.024658 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.024615    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:24.024599999 +0000 UTC m=+9.075404057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:20.125293 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:20.125217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:20.125501 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.125443    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:20.125501 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.125466    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:20.125501 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.125479    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:20.125662 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.125541    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:24.125521714 +0000 UTC m=+9.176325774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:20.456799 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:20.456760    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:20.457019 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.456909    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:20.457412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:20.457391    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:20.457530 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:20.457504    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:21.457387 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:21.457204    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:21.457387 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:21.457364    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:22.457207 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:22.457174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:22.457418 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:22.457306    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:22.457418 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:22.457364    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:22.457913 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:22.457480    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:23.461098 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:23.460473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:23.461098 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:23.460726    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:24.061423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:24.061334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:24.061423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:24.061376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:24.061771 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.061514    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:24.061771 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.061571    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:32.061557943 +0000 UTC m=+17.112361996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:24.061771 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.061621    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:24.061771 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.061651    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:32.061641002 +0000 UTC m=+17.112445060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:24.162889 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:24.162844    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:24.163098 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.163047    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:24.163098 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.163078    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:24.163098 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.163092    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:24.163255 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.163158    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:32.163137044 +0000 UTC m=+17.213941121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:24.457075 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:24.457033    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:24.457293 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:24.457088    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:24.457293 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.457169    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:24.457424 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:24.457353    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:25.458003 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:25.457968    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:25.458479 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:25.458091    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:26.456933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:26.456893    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:26.457106 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:26.457026    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:26.457442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:26.457419    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:26.457542 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:26.457522    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:27.457786 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:27.457295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:27.457786 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:27.457430    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:28.456963 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:28.456902    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:28.457143 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:28.457031    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:28.457143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:28.457089    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:28.457287 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:28.457180    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:29.459736 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:29.457249    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:29.459736 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:29.457382    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:30.457313 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:30.457273    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:30.457313 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:30.457309    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:30.457571 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:30.457403    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:30.457571 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:30.457537    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:31.459796 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:31.459768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:31.460197 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:31.459899    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:32.125295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:32.125256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:32.125295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:32.125302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:32.125583 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.125421    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:32.125583 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.125477    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:48.125463752 +0000 UTC m=+33.176267803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:32.125583 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.125421    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:32.125749 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.125601    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:01:48.125580191 +0000 UTC m=+33.176384246 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:32.226515 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:32.226474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:32.226740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.226642    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:32.226740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.226659    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:32.226740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.226668    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:32.226740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.226730    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:01:48.226711597 +0000 UTC m=+33.277515671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:32.456721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:32.456610    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:32.456721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:32.456667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:32.456937 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.456761    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:32.456937 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:32.456894    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:33.456774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:33.456741    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:33.457303 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:33.456894    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:34.457376 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:34.457337    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:34.457872 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:34.457381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:34.457872 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:34.457483    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:34.457872 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:34.457640    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:35.460450 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:35.460412    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:35.460935 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:35.460541    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:36.456492 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:36.456451    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:36.456492 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:36.456484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:36.456740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:36.456586    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:36.456740 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:36.456710    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:37.457075 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:37.457052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:37.457459 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:37.457200    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:38.457481 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.457250    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:38.458170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.457252    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:38.458170 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:38.457567    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:38.458170 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:38.457716    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:38.570517 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.570480    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-45wh8" event={"ID":"53fa9439-4cdf-4b78-8e90-7bc32b242d54","Type":"ContainerStarted","Data":"3120983df2f5b64e4adf9434398cae52e56f78e64fdac6fd363c8a64b98821e1"}
Jul 28 16:01:38.572336 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.572306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal" event={"ID":"834e29422605a4563abd0c2b2da8dece","Type":"ContainerStarted","Data":"501041b777d497ce574773cfcc19e64709c4d6970005f0fe5d5bfa91c0915fc6"}
Jul 28 16:01:38.575476 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.575456    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:01:38.575840 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.575809    2556 generic.go:356] "Generic (PLEG): container finished" podID="7901a8b4-34c9-4668-b0f9-759fff4e035f" containerID="c32627b757f58c73ae977522ad259dbdee504e5eb64d4c62f7fe8f24fef440ed" exitCode=1
Jul 28 16:01:38.575941 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.575877    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"cafc652d8e5458b0f1feec2901d86e5e03cc28d1b6d9c9850e93e42968c1f5e5"}
Jul 28 16:01:38.575985 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.575949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"052ad84db91568a53a748cf70fa8d57d05c8e567df7f94e30e4a1d5f2129c00f"}
Jul 28 16:01:38.575985 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.575982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"6a7e5fdd8060e35e545c06817be20ef42675603d86a8f6771a8af210228839c0"}
Jul 28 16:01:38.576086 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.576011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"259fc9f6a1ebb8f76eae610352eb6e84e1b71415029aa8cb9a7e5d41ac4791d5"}
Jul 28 16:01:38.576086 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.576037    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerDied","Data":"c32627b757f58c73ae977522ad259dbdee504e5eb64d4c62f7fe8f24fef440ed"}
Jul 28 16:01:38.576086 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.576066    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"a49822a6345f7120497299989e5f5779e7784244cbdd2515d562555a5403edce"}
Jul 28 16:01:38.577718 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.577697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fshgz" event={"ID":"df7f31c1-0c68-4404-bdf1-e4be1580385f","Type":"ContainerStarted","Data":"4a301c73ee61db9e6c13fdd7edadf6f8b3fb30c45715f35059fdb8975c6a5c03"}
Jul 28 16:01:38.623790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.623746    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-45wh8" podStartSLOduration=1.601244233 podStartE2EDuration="22.623727987s" totalImagesPullingTime="21.022483754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:38.596887891 +0000 UTC m=+23.647691965" watchObservedRunningTime="2026-07-28 16:01:38.623727987 +0000 UTC m=+23.674532054"
Jul 28 16:01:38.624404 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:38.624376    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-124.ec2.internal" podStartSLOduration=22.624365664 podStartE2EDuration="22.624365664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:38.623996631 +0000 UTC m=+23.674800703" watchObservedRunningTime="2026-07-28 16:01:38.624365664 +0000 UTC m=+23.675169738"
Jul 28 16:01:39.457335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.457126    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:39.459539 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:39.457912    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:39.581099 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.581014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rcrxp" event={"ID":"811710f4-2c68-43dd-a1df-f3cad45012c1","Type":"ContainerStarted","Data":"3e6774860a9289cfdfde2b2122b11b8f43860d8e45829b4778f95d77d28a9659"}
Jul 28 16:01:39.582492 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.582463    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="e6bc5f42631f2e405a96a2137c60256cdf540f783b9e39c2451469baa36a46a7" exitCode=0
Jul 28 16:01:39.582580 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.582510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"e6bc5f42631f2e405a96a2137c60256cdf540f783b9e39c2451469baa36a46a7"}
Jul 28 16:01:39.584111 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.584042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6j4ql" event={"ID":"edd4ba9f-f16f-42de-ac26-975c51735d1a","Type":"ContainerStarted","Data":"69191da371898bef23a8f827b1ab7fd1f047d7b29957e04e51ee1ed6daa506e1"}
Jul 28 16:01:39.585510 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.585484    2556 generic.go:356] "Generic (PLEG): container finished" podID="35429a64cc41bf0a3ba185325ae3dc4f" containerID="21af5011934e9a7864f4477963c311be4cc67709d3dfae25e1b76812c8c989f6" exitCode=0
Jul 28 16:01:39.585583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.585567    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal" event={"ID":"35429a64cc41bf0a3ba185325ae3dc4f","Type":"ContainerDied","Data":"21af5011934e9a7864f4477963c311be4cc67709d3dfae25e1b76812c8c989f6"}
Jul 28 16:01:39.586991 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.586965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69jt2" event={"ID":"425eac07-3ae4-4c5a-94b6-839025d8d7ed","Type":"ContainerStarted","Data":"3afbb52528b274fdc4d2837f02a5ff0f110ff9f79931a5c299b54edca475b557"}
Jul 28 16:01:39.588800 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.588780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7" event={"ID":"3ea2d5c9-69d3-45e2-b740-888c50866a86","Type":"ContainerStarted","Data":"e832b7331f49918ede298eb0f6783443b3a2f65005df4e1be4267266cb2d0977"}
Jul 28 16:01:39.590056 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.590028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r9jkf" event={"ID":"a3e9014d-d847-44b4-91a3-b6c78730faed","Type":"ContainerStarted","Data":"97351c0d7097945facec633a1ce9da491289cc6bbe0fe635da980937f9e80199"}
Jul 28 16:01:39.615473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.615416    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fshgz" podStartSLOduration=2.621036916 podStartE2EDuration="23.615399835s" totalImagesPullingTime="20.994362919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:38.650657325 +0000 UTC m=+23.701461395" watchObservedRunningTime="2026-07-28 16:01:39.615399835 +0000 UTC m=+24.666203907"
Jul 28 16:01:39.637754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.637702    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6j4ql" podStartSLOduration=3.514838872 podStartE2EDuration="23.637682952s" totalImagesPullingTime="20.12284408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:39.637615767 +0000 UTC m=+24.688419841" watchObservedRunningTime="2026-07-28 16:01:39.637682952 +0000 UTC m=+24.688487026"
Jul 28 16:01:39.638129 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.638099    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rcrxp" podStartSLOduration=3.568503745 podStartE2EDuration="23.638090429s" totalImagesPullingTime="20.069586684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:39.615659218 +0000 UTC m=+24.666463290" watchObservedRunningTime="2026-07-28 16:01:39.638090429 +0000 UTC m=+24.688894504"
Jul 28 16:01:39.797463 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.797412    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-69jt2" podStartSLOduration=2.783867234 podStartE2EDuration="23.797396134s" totalImagesPullingTime="21.0135289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:39.671308243 +0000 UTC m=+24.722112315" watchObservedRunningTime="2026-07-28 16:01:39.797396134 +0000 UTC m=+24.848200207"
Jul 28 16:01:39.859111 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:39.859058    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r9jkf" podStartSLOduration=2.8940208419999998 podStartE2EDuration="23.859041722s" totalImagesPullingTime="20.96502088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:39.858587548 +0000 UTC m=+24.909391620" watchObservedRunningTime="2026-07-28 16:01:39.859041722 +0000 UTC m=+24.909845835"
Jul 28 16:01:40.008259 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.008230    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 16:01:40.352971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.352893    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T16:01:40.008251123Z","UUID":"45de8bae-22d6-4aa8-bcdd-1b43c0ead0c6","Handler":null,"Name":"","Endpoint":""}
Jul 28 16:01:40.354316 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.354298    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 16:01:40.354426 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.354327    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 16:01:40.457220 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.457191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:40.457220 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.457219    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:40.457465 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:40.457317    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:40.457465 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:40.457413    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:40.595065 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.595030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal" event={"ID":"35429a64cc41bf0a3ba185325ae3dc4f","Type":"ContainerStarted","Data":"37bd63380432237dbf0dc64e8f95b5626fa3f3f0f19277010e50ab027fcb5acf"}
Jul 28 16:01:40.597507 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.597151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7" event={"ID":"3ea2d5c9-69d3-45e2-b740-888c50866a86","Type":"ContainerStarted","Data":"5170bf5da36fd695919526f82338083565246e8aaa0476268ec9155b3f41ef4e"}
Jul 28 16:01:40.611003 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:40.610910    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-124.ec2.internal" podStartSLOduration=24.610895393 podStartE2EDuration="24.610895393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:40.61086215 +0000 UTC m=+25.661666220" watchObservedRunningTime="2026-07-28 16:01:40.610895393 +0000 UTC m=+25.661699466"
Jul 28 16:01:41.238872 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.238778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:41.239362 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.239344    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:41.457039 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.457004    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:41.457216 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:41.457118    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:41.602249 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.602222    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:01:41.602749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.602640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"9b88fb4b75742b88c4e246d8533fc4556b5d17d279aa7d708ea35df96cdb5700"}
Jul 28 16:01:41.604469 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.604440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7" event={"ID":"3ea2d5c9-69d3-45e2-b740-888c50866a86","Type":"ContainerStarted","Data":"7860185b5acb33f46725579d6b72b804074916e86e31c8a5973549827b6ab0a0"}
Jul 28 16:01:41.604768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.604753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:41.605204 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.605188    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rcrxp"
Jul 28 16:01:41.632732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:41.632679    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbfm7" podStartSLOduration=2.395725238 podStartE2EDuration="25.632659496s" totalImagesPullingTime="23.236934258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:41.631969482 +0000 UTC m=+26.682773555" watchObservedRunningTime="2026-07-28 16:01:41.632659496 +0000 UTC m=+26.683463580"
Jul 28 16:01:42.456857 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:42.456825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:42.457064 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:42.456893    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:42.457064 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:42.456992    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:42.457305 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:42.457275    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:43.456810 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:43.456567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:43.457424 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:43.456920    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:43.613572 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:43.613534    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:01:43.614081 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:43.614039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"f1dc2e0c7a85b582ae5e7f16075daf52b5b8f5fc53757b59b1d8fc3ad4ffeabd"}
Jul 28 16:01:43.614467 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:43.614450    2556 scope.go:117] "RemoveContainer" containerID="c32627b757f58c73ae977522ad259dbdee504e5eb64d4c62f7fe8f24fef440ed"
Jul 28 16:01:44.456898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.456434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:44.456898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.456448    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:44.456898 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:44.456542    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:44.456898 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:44.456643    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:44.620890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.620855    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:01:44.621288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.621262    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" event={"ID":"7901a8b4-34c9-4668-b0f9-759fff4e035f","Type":"ContainerStarted","Data":"33d8559ac9da71687ff79d8f4dc6ae305a0380079bff9175ef99e142ac719e52"}
Jul 28 16:01:44.621705 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.621676    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:44.621829 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.621756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:44.621829 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.621802    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:44.640142 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.640111    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:44.641897 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.641870    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:01:44.654206 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:44.654149    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q" podStartSLOduration=7.69190735 podStartE2EDuration="28.65412846s" totalImagesPullingTime="20.96222111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:44.653272892 +0000 UTC m=+29.704076993" watchObservedRunningTime="2026-07-28 16:01:44.65412846 +0000 UTC m=+29.704932533"
Jul 28 16:01:45.457909 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:45.457867    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:45.458330 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:45.457997    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:46.457122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:46.456932    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:46.457406 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:46.456932    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:46.457406 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:46.457199    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:46.457406 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:46.457291    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:46.628153 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:46.628123    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="1f5568cce75a97197753db872775e0bc055a3d9fbe2c51b3c500b0d65b130ea3" exitCode=0
Jul 28 16:01:46.628547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:46.628210    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"1f5568cce75a97197753db872775e0bc055a3d9fbe2c51b3c500b0d65b130ea3"}
Jul 28 16:01:47.457281 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:47.457248    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:47.457456 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:47.457393    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:47.632493 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:47.632459    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="39c21fa64bd3c066afb96d2b1ba838207172deb76f6cb15ca0d76a0dbf4151fc" exitCode=0
Jul 28 16:01:47.632976 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:47.632519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"39c21fa64bd3c066afb96d2b1ba838207172deb76f6cb15ca0d76a0dbf4151fc"}
Jul 28 16:01:48.138992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:48.138955    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:48.138992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:48.138996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:48.139228 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.139093    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:48.139228 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.139096    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:48.139228 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.139147    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs podName:9da7f48a-38cf-4878-bb8d-53c58670b5d7 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:20.139132632 +0000 UTC m=+65.189936683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs") pod "network-metrics-daemon-v2hxs" (UID: "9da7f48a-38cf-4878-bb8d-53c58670b5d7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 16:01:48.139228 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.139160    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret podName:17c73738-c4bf-40fc-ad8c-7bac3ad2db11 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:20.13915382 +0000 UTC m=+65.189957871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret") pod "global-pull-secret-syncer-j225x" (UID: "17c73738-c4bf-40fc-ad8c-7bac3ad2db11") : object "kube-system"/"original-pull-secret" not registered
Jul 28 16:01:48.239823 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:48.239791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:48.240150 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.239952    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 16:01:48.240150 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.239974    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 16:01:48.240150 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.239982    2556 projected.go:196] Error preparing data for projected volume kube-api-access-4pbr9 for pod openshift-network-diagnostics/network-check-target-bfrcj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:48.240150 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.240034    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9 podName:a4e33799-9ac4-4628-a0fb-170ceda351bb nodeName:}" failed. No retries permitted until 2026-07-28 16:02:20.240019704 +0000 UTC m=+65.290823755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4pbr9" (UniqueName: "kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9") pod "network-check-target-bfrcj" (UID: "a4e33799-9ac4-4628-a0fb-170ceda351bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 16:01:48.456831 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:48.456741    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:48.456831 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:48.456768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:48.457017 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.456864    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:48.457017 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:48.456948    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:49.457393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:49.457360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:49.457805 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:49.457469    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:50.456413 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:50.456375    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:50.456594 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:50.456380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:50.456594 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:50.456479    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:50.456594 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:50.456552    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:51.457439 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:51.457353    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:51.457803 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:51.457482    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:51.643840 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:51.643802    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="8169141d550caaa1bd5f7040753b792594314e01a656eb8fe992cb8952ef3a4e" exitCode=0
Jul 28 16:01:51.644066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:51.643858    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"8169141d550caaa1bd5f7040753b792594314e01a656eb8fe992cb8952ef3a4e"}
Jul 28 16:01:52.457071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:52.457029    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:52.457071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:52.457071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:52.457316 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:52.457160    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:52.457316 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:52.457269    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:53.456696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:53.456664    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:53.457169 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:53.456843    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:54.457063 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:54.457028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:54.457540 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:54.457043    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:54.457540 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:54.457177    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:54.457540 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:54.457251    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:55.457894 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:55.457853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:55.458382 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:55.457994    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:56.456531 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:56.456492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:56.456726 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:56.456492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:56.456726 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:56.456602    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:56.456726 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:56.456715    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:57.457017 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:57.456980    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:57.457501 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:57.457124    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:57.661002 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:57.660970    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="1c4b4fac78e3b60cae46c23239580adb2c5c89956333b177255bdced34473804" exitCode=0
Jul 28 16:01:57.661162 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:57.661008    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"1c4b4fac78e3b60cae46c23239580adb2c5c89956333b177255bdced34473804"}
Jul 28 16:01:58.456421 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:58.456378    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:01:58.456599 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:58.456483    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:01:58.456599 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:58.456534    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:01:58.456704 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:58.456617    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:01:58.666349 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:58.666313    2556 generic.go:356] "Generic (PLEG): container finished" podID="28ef4649-bfed-43a0-aabc-301174665b34" containerID="4559eef84afc018de1abbd5b85b6819e58bb670c08cf81953514473a564af586" exitCode=0
Jul 28 16:01:58.666742 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:58.666358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerDied","Data":"4559eef84afc018de1abbd5b85b6819e58bb670c08cf81953514473a564af586"}
Jul 28 16:01:59.457027 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:59.456989    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:01:59.457202 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:01:59.457112    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:01:59.671956 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:59.671917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w544q" event={"ID":"28ef4649-bfed-43a0-aabc-301174665b34","Type":"ContainerStarted","Data":"b588e54f91a66350903af8014c993948b9117d9cb8a277cd6c9eded8d22b791a"}
Jul 28 16:01:59.700284 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:01:59.700236    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w544q" podStartSLOduration=7.138728395 podStartE2EDuration="43.700220115s" totalImagesPullingTime="36.56149172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:01:59.699003426 +0000 UTC m=+44.749807498" watchObservedRunningTime="2026-07-28 16:01:59.700220115 +0000 UTC m=+44.751024184"
Jul 28 16:02:00.456660 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:00.456607    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:00.456832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:00.456607    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:00.456832 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:00.456732    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:02:00.456832 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:00.456798    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:02:01.457415 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:01.457379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:01.457821 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:01.457495    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:02:02.457214 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:02.457166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:02.457647 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:02.457196    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:02.458134 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:02.457691    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:02:02.458279 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:02.458258    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:02:03.457092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.456908    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:03.457267 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:03.457180    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:02:03.526833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.526803    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bfrcj"]
Jul 28 16:02:03.527396 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.526935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:03.527396 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:03.527044    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:02:03.531940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.531869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j225x"]
Jul 28 16:02:03.532075 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.531956    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:03.532075 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:03.532036    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:02:03.533890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.533148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2hxs"]
Jul 28 16:02:03.680353 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:03.680321    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:03.680521 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:03.680441    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:02:05.457529 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:05.457496    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:05.458048 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:05.457567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:05.458048 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:05.457579    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:02:05.458048 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:05.457684    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:02:05.458048 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:05.457734    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:05.458048 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:05.457791    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:02:07.456965 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.456935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:07.457451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.456935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:07.457451 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:07.457069    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bfrcj" podUID="a4e33799-9ac4-4628-a0fb-170ceda351bb"
Jul 28 16:02:07.457451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.456933    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:07.457451 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:07.457122    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j225x" podUID="17c73738-c4bf-40fc-ad8c-7bac3ad2db11"
Jul 28 16:02:07.457451 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:07.457250    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2hxs" podUID="9da7f48a-38cf-4878-bb8d-53c58670b5d7"
Jul 28 16:02:07.783107 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.783038    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-124.ec2.internal" event="NodeReady"
Jul 28 16:02:07.783253 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.783163    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 16:02:07.850680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.849533    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j8c2p"]
Jul 28 16:02:07.868441 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.868403    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-62m44"]
Jul 28 16:02:07.868637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.868557    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:07.871223 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.871198    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 16:02:07.871377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.871363    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vc5px\""
Jul 28 16:02:07.871432 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.871401    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 16:02:07.892642 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.892585    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8c2p"]
Jul 28 16:02:07.892811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.892652    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q7jgs"]
Jul 28 16:02:07.892811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.892716    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:07.896663 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.896610    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 16:02:07.896850 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.896831    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fswjp\""
Jul 28 16:02:07.896970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.896950    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 16:02:07.897028 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.896962    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 16:02:07.906597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.906564    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62m44"]
Jul 28 16:02:07.906597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.906604    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q7jgs"]
Jul 28 16:02:07.906839 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.906720    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:07.910378 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.910357    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 16:02:07.910467 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.910379    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 16:02:07.910672 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.910658    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 16:02:07.910746 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.910673    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bxxbb\""
Jul 28 16:02:07.910789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.910676    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 16:02:07.987559 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987516    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7324c4bf-48ba-47b3-8819-83fde99ff606-config-volume\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:07.987768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987597    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7324c4bf-48ba-47b3-8819-83fde99ff606-tmp-dir\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:07.987768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987647    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzfv\" (UniqueName: \"kubernetes.io/projected/7324c4bf-48ba-47b3-8819-83fde99ff606-kube-api-access-8gzfv\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:07.987768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-cert\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:07.987768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987722    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7324c4bf-48ba-47b3-8819-83fde99ff606-metrics-tls\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:07.987768 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:07.987765    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8h6\" (UniqueName: \"kubernetes.io/projected/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-kube-api-access-sb8h6\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.088938 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.088850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-data-volume\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.088938 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.088921    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzhw\" (UniqueName: \"kubernetes.io/projected/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-api-access-nmzhw\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.088943    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7324c4bf-48ba-47b3-8819-83fde99ff606-tmp-dir\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.088969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzfv\" (UniqueName: \"kubernetes.io/projected/7324c4bf-48ba-47b3-8819-83fde99ff606-kube-api-access-8gzfv\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.088987    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-cert\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089002    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7324c4bf-48ba-47b3-8819-83fde99ff606-metrics-tls\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8h6\" (UniqueName: \"kubernetes.io/projected/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-kube-api-access-sb8h6\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.089155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7324c4bf-48ba-47b3-8819-83fde99ff606-config-volume\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.089455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-crio-socket\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.089455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.089455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089342    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7324c4bf-48ba-47b3-8819-83fde99ff606-tmp-dir\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.089723 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.089699    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7324c4bf-48ba-47b3-8819-83fde99ff606-config-volume\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.093202 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.093173    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7324c4bf-48ba-47b3-8819-83fde99ff606-metrics-tls\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.093320 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.093278    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-cert\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.098743 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.098718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzfv\" (UniqueName: \"kubernetes.io/projected/7324c4bf-48ba-47b3-8819-83fde99ff606-kube-api-access-8gzfv\") pod \"dns-default-j8c2p\" (UID: \"7324c4bf-48ba-47b3-8819-83fde99ff606\") " pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.098860 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.098740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8h6\" (UniqueName: \"kubernetes.io/projected/c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e-kube-api-access-sb8h6\") pod \"ingress-canary-62m44\" (UID: \"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e\") " pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.178278 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.178234    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:08.190542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190500    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-crio-socket\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.190542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190540    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.190789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-data-volume\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.190789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzhw\" (UniqueName: \"kubernetes.io/projected/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-api-access-nmzhw\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.190789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.190910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.190788    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-crio-socket\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.191105 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.191083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-data-volume\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.191288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.191273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.193149 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.193133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.202102 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.202068    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-62m44"
Jul 28 16:02:08.202330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.202315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzhw\" (UniqueName: \"kubernetes.io/projected/458c263d-9f6a-4188-ac89-0eed7b1a8d7b-kube-api-access-nmzhw\") pod \"insights-runtime-extractor-q7jgs\" (UID: \"458c263d-9f6a-4188-ac89-0eed7b1a8d7b\") " pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.215293 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.215258    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q7jgs"
Jul 28 16:02:08.412262 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.412081    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-62m44"]
Jul 28 16:02:08.422289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.422258    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8c2p"]
Jul 28 16:02:08.442781 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:08.442742    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458c263d_9f6a_4188_ac89_0eed7b1a8d7b.slice/crio-ed3ebe6ec855bbd8677ba02c860de5d7ca67123dc7bde133b47550f78f1ad7c4 WatchSource:0}: Error finding container ed3ebe6ec855bbd8677ba02c860de5d7ca67123dc7bde133b47550f78f1ad7c4: Status 404 returned error can't find the container with id ed3ebe6ec855bbd8677ba02c860de5d7ca67123dc7bde133b47550f78f1ad7c4
Jul 28 16:02:08.442963 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.442863    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q7jgs"]
Jul 28 16:02:08.693196 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.693103    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7jgs" event={"ID":"458c263d-9f6a-4188-ac89-0eed7b1a8d7b","Type":"ContainerStarted","Data":"ed3ebe6ec855bbd8677ba02c860de5d7ca67123dc7bde133b47550f78f1ad7c4"}
Jul 28 16:02:08.694323 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.694292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62m44" event={"ID":"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e","Type":"ContainerStarted","Data":"6bf76b40327f3ca17bd1c2e085b3c58c05880e2ce8a1cd3e415ccd2b683b7a2f"}
Jul 28 16:02:08.695319 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:08.695297    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8c2p" event={"ID":"7324c4bf-48ba-47b3-8819-83fde99ff606","Type":"ContainerStarted","Data":"0dc8bbbff55069fb624e2eace740f64fee258159d42e8fa9c2ae8f7e951673bc"}
Jul 28 16:02:09.457874 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.457839    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:09.460279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.458527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:09.460279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.458735    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:09.461335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.461307    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:02:09.462838 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.462816    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:02:09.463045 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.463027    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:02:09.463308 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.463278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kkdj2\""
Jul 28 16:02:09.463500 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.463483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:02:09.463733 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:09.463689    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84r49\""
Jul 28 16:02:11.707417 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.707325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-62m44" event={"ID":"c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e","Type":"ContainerStarted","Data":"720c4f41c7179d629d9591ff9ef566751264d91fdcfe7f499d02434d5a6a08a9"}
Jul 28 16:02:11.710010 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.709971    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8c2p" event={"ID":"7324c4bf-48ba-47b3-8819-83fde99ff606","Type":"ContainerStarted","Data":"10314f68f17c95bfa968bf25b507b3cf14bc0ec2419514bf1eaf4be73ab09406"}
Jul 28 16:02:11.710147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.710028    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:11.710147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.710046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8c2p" event={"ID":"7324c4bf-48ba-47b3-8819-83fde99ff606","Type":"ContainerStarted","Data":"2460a1005ff3a12fe8e3cda79c98aea04d2151094d171baded4daac05e09b9b8"}
Jul 28 16:02:11.733098 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.733037    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-62m44" podStartSLOduration=1.786474696 podStartE2EDuration="4.733019571s" totalImagesPullingTime="2.946544875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:11.732680132 +0000 UTC m=+56.783484205" watchObservedRunningTime="2026-07-28 16:02:11.733019571 +0000 UTC m=+56.783823626"
Jul 28 16:02:11.755089 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:11.755021    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j8c2p" podStartSLOduration=1.809503525 podStartE2EDuration="4.75499927s" totalImagesPullingTime="2.945495745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:11.753762656 +0000 UTC m=+56.804566730" watchObservedRunningTime="2026-07-28 16:02:11.75499927 +0000 UTC m=+56.805803343"
Jul 28 16:02:12.793890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.793854    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-v5vhn"]
Jul 28 16:02:12.808644 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.808585    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:12.812748 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.812720    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 16:02:12.812901 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.812720    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t2gc9\""
Jul 28 16:02:12.812901 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.812872    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 16:02:12.819001 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.818970    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-v5vhn"]
Jul 28 16:02:12.826150 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.826120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfk8\" (UniqueName: \"kubernetes.io/projected/45254f28-7160-4e38-b5d9-ef3d1aad5181-kube-api-access-ctfk8\") pod \"downloads-7c97dc4555-v5vhn\" (UID: \"45254f28-7160-4e38-b5d9-ef3d1aad5181\") " pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:12.927070 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.927028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfk8\" (UniqueName: \"kubernetes.io/projected/45254f28-7160-4e38-b5d9-ef3d1aad5181-kube-api-access-ctfk8\") pod \"downloads-7c97dc4555-v5vhn\" (UID: \"45254f28-7160-4e38-b5d9-ef3d1aad5181\") " pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:12.939129 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:12.939071    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfk8\" (UniqueName: \"kubernetes.io/projected/45254f28-7160-4e38-b5d9-ef3d1aad5181-kube-api-access-ctfk8\") pod \"downloads-7c97dc4555-v5vhn\" (UID: \"45254f28-7160-4e38-b5d9-ef3d1aad5181\") " pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:13.119298 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:13.119249    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:14.473824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:14.473785    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-v5vhn"]
Jul 28 16:02:14.478706 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:14.478663    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45254f28_7160_4e38_b5d9_ef3d1aad5181.slice/crio-16df4c7cbab61f680918653fcf12c51ff5f8f69551ac5d32b6d62d8a81a0fa2d WatchSource:0}: Error finding container 16df4c7cbab61f680918653fcf12c51ff5f8f69551ac5d32b6d62d8a81a0fa2d: Status 404 returned error can't find the container with id 16df4c7cbab61f680918653fcf12c51ff5f8f69551ac5d32b6d62d8a81a0fa2d
Jul 28 16:02:14.721098 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:14.720999    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7jgs" event={"ID":"458c263d-9f6a-4188-ac89-0eed7b1a8d7b","Type":"ContainerStarted","Data":"4ba3077c0084d174bcc47fb027dfd8b14d8ea423019990c3e6deae257dc95fbb"}
Jul 28 16:02:14.722160 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:14.722135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-v5vhn" event={"ID":"45254f28-7160-4e38-b5d9-ef3d1aad5181","Type":"ContainerStarted","Data":"16df4c7cbab61f680918653fcf12c51ff5f8f69551ac5d32b6d62d8a81a0fa2d"}
Jul 28 16:02:16.651014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:16.650982    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mlq9q"
Jul 28 16:02:17.733939 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.733903    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"]
Jul 28 16:02:17.752537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.752503    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.752725 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.752567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"]
Jul 28 16:02:17.757367 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.757322    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 16:02:17.757535 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.757405    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fcjfd\""
Jul 28 16:02:17.757895 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.757875    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 16:02:17.758326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.758302    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 16:02:17.758432 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.758347    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 16:02:17.763802 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.763773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjm2q\" (UniqueName: \"kubernetes.io/projected/706bc98f-b212-4795-a188-646c7c73a311-kube-api-access-cjm2q\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.763933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.763839    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.763933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.763876    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.764031 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.763948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706bc98f-b212-4795-a188-646c7c73a311-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.766897 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.766872    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 16:02:17.864749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.864699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjm2q\" (UniqueName: \"kubernetes.io/projected/706bc98f-b212-4795-a188-646c7c73a311-kube-api-access-cjm2q\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.864940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.864790    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.864940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.864818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.864940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.864849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706bc98f-b212-4795-a188-646c7c73a311-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.864940 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:17.864918    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 16:02:17.865093 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:17.864981    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls podName:706bc98f-b212-4795-a188-646c7c73a311 nodeName:}" failed. No retries permitted until 2026-07-28 16:02:18.364966566 +0000 UTC m=+63.415770619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-cgcsn" (UID: "706bc98f-b212-4795-a188-646c7c73a311") : secret "prometheus-operator-tls" not found
Jul 28 16:02:17.865527 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.865501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/706bc98f-b212-4795-a188-646c7c73a311-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.867923 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.867896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:17.875551 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:17.875496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjm2q\" (UniqueName: \"kubernetes.io/projected/706bc98f-b212-4795-a188-646c7c73a311-kube-api-access-cjm2q\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:18.368754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:18.368709    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:18.374483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:18.374450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/706bc98f-b212-4795-a188-646c7c73a311-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-cgcsn\" (UID: \"706bc98f-b212-4795-a188-646c7c73a311\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:18.664893 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:18.664858    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"
Jul 28 16:02:18.742532 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:18.742485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7jgs" event={"ID":"458c263d-9f6a-4188-ac89-0eed7b1a8d7b","Type":"ContainerStarted","Data":"4eb9bf502af16165069d073ad1f861f9f51603b80b158111119b0ad321fdc84a"}
Jul 28 16:02:18.821548 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:18.821512    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-cgcsn"]
Jul 28 16:02:18.825483 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:18.825383    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706bc98f_b212_4795_a188_646c7c73a311.slice/crio-c82a8d46f8ff5a0d95b98a96617172cafa8842069955d2d595dfa1f075373f88 WatchSource:0}: Error finding container c82a8d46f8ff5a0d95b98a96617172cafa8842069955d2d595dfa1f075373f88: Status 404 returned error can't find the container with id c82a8d46f8ff5a0d95b98a96617172cafa8842069955d2d595dfa1f075373f88
Jul 28 16:02:19.749946 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:19.749863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn" event={"ID":"706bc98f-b212-4795-a188-646c7c73a311","Type":"ContainerStarted","Data":"c82a8d46f8ff5a0d95b98a96617172cafa8842069955d2d595dfa1f075373f88"}
Jul 28 16:02:20.185758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.185718    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:20.185758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.185767    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:20.189541 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.189511    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 16:02:20.190326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.190299    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 16:02:20.199991 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.199964    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9da7f48a-38cf-4878-bb8d-53c58670b5d7-metrics-certs\") pod \"network-metrics-daemon-v2hxs\" (UID: \"9da7f48a-38cf-4878-bb8d-53c58670b5d7\") " pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:20.203251 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.203193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17c73738-c4bf-40fc-ad8c-7bac3ad2db11-original-pull-secret\") pod \"global-pull-secret-syncer-j225x\" (UID: \"17c73738-c4bf-40fc-ad8c-7bac3ad2db11\") " pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:20.285407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.285369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j225x"
Jul 28 16:02:20.287094 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.287063    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:20.290851 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.290825    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 16:02:20.300993 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.300788    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 16:02:20.309264 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.309170    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kkdj2\""
Jul 28 16:02:20.312451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.312398    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbr9\" (UniqueName: \"kubernetes.io/projected/a4e33799-9ac4-4628-a0fb-170ceda351bb-kube-api-access-4pbr9\") pod \"network-check-target-bfrcj\" (UID: \"a4e33799-9ac4-4628-a0fb-170ceda351bb\") " pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:20.317345 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.317115    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2hxs"
Jul 28 16:02:20.317955 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.317743    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84r49\""
Jul 28 16:02:20.326412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.326191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:20.565654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.565596    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j225x"]
Jul 28 16:02:20.573989 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:20.573885    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c73738_c4bf_40fc_ad8c_7bac3ad2db11.slice/crio-2b51addf43c68e27a93d95c53d910e71e7d505595b2cfacbeb09cf9c3750ce75 WatchSource:0}: Error finding container 2b51addf43c68e27a93d95c53d910e71e7d505595b2cfacbeb09cf9c3750ce75: Status 404 returned error can't find the container with id 2b51addf43c68e27a93d95c53d910e71e7d505595b2cfacbeb09cf9c3750ce75
Jul 28 16:02:20.587490 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.587457    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bfrcj"]
Jul 28 16:02:20.592823 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:20.592800    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e33799_9ac4_4628_a0fb_170ceda351bb.slice/crio-2ac76b2371915d747c8c75e2e047ade5e6432d16c65fa43aa184ccf64281b6bc WatchSource:0}: Error finding container 2ac76b2371915d747c8c75e2e047ade5e6432d16c65fa43aa184ccf64281b6bc: Status 404 returned error can't find the container with id 2ac76b2371915d747c8c75e2e047ade5e6432d16c65fa43aa184ccf64281b6bc
Jul 28 16:02:20.611600 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.611572    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2hxs"]
Jul 28 16:02:20.615565 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:20.615530    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da7f48a_38cf_4878_bb8d_53c58670b5d7.slice/crio-295c72f94d5e1b8f133b885d365c2e49fcb097f212a879c52864fcc37dd557c3 WatchSource:0}: Error finding container 295c72f94d5e1b8f133b885d365c2e49fcb097f212a879c52864fcc37dd557c3: Status 404 returned error can't find the container with id 295c72f94d5e1b8f133b885d365c2e49fcb097f212a879c52864fcc37dd557c3
Jul 28 16:02:20.759012 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.758907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bfrcj" event={"ID":"a4e33799-9ac4-4628-a0fb-170ceda351bb","Type":"ContainerStarted","Data":"2ac76b2371915d747c8c75e2e047ade5e6432d16c65fa43aa184ccf64281b6bc"}
Jul 28 16:02:20.760981 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.760946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j225x" event={"ID":"17c73738-c4bf-40fc-ad8c-7bac3ad2db11","Type":"ContainerStarted","Data":"2b51addf43c68e27a93d95c53d910e71e7d505595b2cfacbeb09cf9c3750ce75"}
Jul 28 16:02:20.765538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.765504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn" event={"ID":"706bc98f-b212-4795-a188-646c7c73a311","Type":"ContainerStarted","Data":"26d03d17b1159ff9b3a7f615377e8692dc96400de65a46c88cb576ef319d601d"}
Jul 28 16:02:20.765728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.765562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn" event={"ID":"706bc98f-b212-4795-a188-646c7c73a311","Type":"ContainerStarted","Data":"bb66580d867673041638378c59cb622beb3f10c9b78bfe767c7ad4cc34d8afe6"}
Jul 28 16:02:20.767712 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.767672    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2hxs" event={"ID":"9da7f48a-38cf-4878-bb8d-53c58670b5d7","Type":"ContainerStarted","Data":"295c72f94d5e1b8f133b885d365c2e49fcb097f212a879c52864fcc37dd557c3"}
Jul 28 16:02:20.784851 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:20.784788    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-cgcsn" podStartSLOduration=2.198627196 podStartE2EDuration="3.784768556s" totalImagesPullingTime="1.58614136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:20.784266824 +0000 UTC m=+65.835070908" watchObservedRunningTime="2026-07-28 16:02:20.784768556 +0000 UTC m=+65.835572614"
Jul 28 16:02:21.716947 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:21.716911    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j8c2p"
Jul 28 16:02:23.204687 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.204066    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"]
Jul 28 16:02:23.207688 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.207662    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.213449 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.213418    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rp4h9\""
Jul 28 16:02:23.213813 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.213792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 16:02:23.214391 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.214378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 16:02:23.219972 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.219890    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"]
Jul 28 16:02:23.280070 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.280034    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w746b"]
Jul 28 16:02:23.283213 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.283164    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"]
Jul 28 16:02:23.284462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.284442    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.287521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.287505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.287756 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.287736    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 16:02:23.287891 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.287861    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 16:02:23.288016 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.287989    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 16:02:23.288279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.288260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vkdgx\""
Jul 28 16:02:23.292061 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.292036    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 16:02:23.292382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.292364    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7htfh\""
Jul 28 16:02:23.292529 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.292400    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 16:02:23.292813 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.292654    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 16:02:23.306890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.306818    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"]
Jul 28 16:02:23.309109 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-root\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309151    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.309277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309183    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.309277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309216    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8j5\" (UniqueName: \"kubernetes.io/projected/512d7939-27c8-435a-a509-d43fa1179b76-kube-api-access-rv8j5\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.309277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/512d7939-27c8-435a-a509-d43fa1179b76-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.309277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-metrics-client-ca\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46494030-ddac-40b4-9370-e89ae7b3102e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-accelerators-collector-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-tls\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309365    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-wtmp\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-textfile\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj98m\" (UniqueName: \"kubernetes.io/projected/4bfcea3d-07d4-435e-80a2-7b3787f35f04-kube-api-access-gj98m\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.309537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.310020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrs2\" (UniqueName: \"kubernetes.io/projected/46494030-ddac-40b4-9370-e89ae7b3102e-kube-api-access-xdrs2\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.310020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.310020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.309638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-sys\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410548    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-textfile\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj98m\" (UniqueName: \"kubernetes.io/projected/4bfcea3d-07d4-435e-80a2-7b3787f35f04-kube-api-access-gj98m\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrs2\" (UniqueName: \"kubernetes.io/projected/46494030-ddac-40b4-9370-e89ae7b3102e-kube-api-access-xdrs2\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-sys\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-root\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8j5\" (UniqueName: \"kubernetes.io/projected/512d7939-27c8-435a-a509-d43fa1179b76-kube-api-access-rv8j5\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.410854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/512d7939-27c8-435a-a509-d43fa1179b76-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-metrics-client-ca\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46494030-ddac-40b4-9370-e89ae7b3102e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-accelerators-collector-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-tls\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.410998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-wtmp\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.411023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.411052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.411301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.411081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.412457 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.412429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-accelerators-collector-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.412559 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.412438    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-textfile\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.413031 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.413002    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/512d7939-27c8-435a-a509-d43fa1179b76-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.413259 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.413236    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-wtmp\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.414678 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.413591    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46494030-ddac-40b4-9370-e89ae7b3102e-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.414678 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.414077    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bfcea3d-07d4-435e-80a2-7b3787f35f04-metrics-client-ca\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.414678 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.414150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-sys\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.414923 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.414763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.414923 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.414831    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bfcea3d-07d4-435e-80a2-7b3787f35f04-root\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.415571 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.415518    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.419179 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.419148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.422225 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.422194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.423089 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.422939    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj98m\" (UniqueName: \"kubernetes.io/projected/4bfcea3d-07d4-435e-80a2-7b3787f35f04-kube-api-access-gj98m\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.423217 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.423204    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/512d7939-27c8-435a-a509-d43fa1179b76-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.426226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.426164    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrs2\" (UniqueName: \"kubernetes.io/projected/46494030-ddac-40b4-9370-e89ae7b3102e-kube-api-access-xdrs2\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.427498 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.427472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.427757 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.427732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46494030-ddac-40b4-9370-e89ae7b3102e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-p25nd\" (UID: \"46494030-ddac-40b4-9370-e89ae7b3102e\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.427945 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.427925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bfcea3d-07d4-435e-80a2-7b3787f35f04-node-exporter-tls\") pod \"node-exporter-w746b\" (UID: \"4bfcea3d-07d4-435e-80a2-7b3787f35f04\") " pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.429540 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.429515    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8j5\" (UniqueName: \"kubernetes.io/projected/512d7939-27c8-435a-a509-d43fa1179b76-kube-api-access-rv8j5\") pod \"kube-state-metrics-56d9f944b4-jsffh\" (UID: \"512d7939-27c8-435a-a509-d43fa1179b76\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.530393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.530307    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"
Jul 28 16:02:23.601173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.601134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w746b"
Jul 28 16:02:23.608337 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.608303    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"
Jul 28 16:02:23.718976 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:23.718935    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfcea3d_07d4_435e_80a2_7b3787f35f04.slice/crio-31f20848c8a146b9f0ddea3a0e645d64232090205e2a519ede4f448df52038f8 WatchSource:0}: Error finding container 31f20848c8a146b9f0ddea3a0e645d64232090205e2a519ede4f448df52038f8: Status 404 returned error can't find the container with id 31f20848c8a146b9f0ddea3a0e645d64232090205e2a519ede4f448df52038f8
Jul 28 16:02:23.789054 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.788158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q7jgs" event={"ID":"458c263d-9f6a-4188-ac89-0eed7b1a8d7b","Type":"ContainerStarted","Data":"410119f277a55b9305ad9e87a1497cf6bedb637f6754a0fb5e331ecbce95f7c2"}
Jul 28 16:02:23.794521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.794405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w746b" event={"ID":"4bfcea3d-07d4-435e-80a2-7b3787f35f04","Type":"ContainerStarted","Data":"31f20848c8a146b9f0ddea3a0e645d64232090205e2a519ede4f448df52038f8"}
Jul 28 16:02:23.810657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.809768    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q7jgs" podStartSLOduration=2.485357634 podStartE2EDuration="16.809748505s" totalImagesPullingTime="14.324390871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:23.809296562 +0000 UTC m=+68.860100649" watchObservedRunningTime="2026-07-28 16:02:23.809748505 +0000 UTC m=+68.860552560"
Jul 28 16:02:23.918820 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.918785    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd"]
Jul 28 16:02:23.924289 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:23.924259    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46494030_ddac_40b4_9370_e89ae7b3102e.slice/crio-9940de1acabdfbe91da79852a95f959c095d4b5828e29e3ad8789e8eac6bb5db WatchSource:0}: Error finding container 9940de1acabdfbe91da79852a95f959c095d4b5828e29e3ad8789e8eac6bb5db: Status 404 returned error can't find the container with id 9940de1acabdfbe91da79852a95f959c095d4b5828e29e3ad8789e8eac6bb5db
Jul 28 16:02:23.944291 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:23.944216    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh"]
Jul 28 16:02:23.953533 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:23.953500    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d7939_27c8_435a_a509_d43fa1179b76.slice/crio-c281ea212c98fc6b6d480ea8c7d4092ba7a7985edf2b5a51cf09a057af272daf WatchSource:0}: Error finding container c281ea212c98fc6b6d480ea8c7d4092ba7a7985edf2b5a51cf09a057af272daf: Status 404 returned error can't find the container with id c281ea212c98fc6b6d480ea8c7d4092ba7a7985edf2b5a51cf09a057af272daf
Jul 28 16:02:24.815747 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.815706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd" event={"ID":"46494030-ddac-40b4-9370-e89ae7b3102e","Type":"ContainerStarted","Data":"269156e783abba1896a3d81ab3fb74bc50739fb300842239ec14a1df347492f9"}
Jul 28 16:02:24.816257 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.815800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd" event={"ID":"46494030-ddac-40b4-9370-e89ae7b3102e","Type":"ContainerStarted","Data":"0b967743b48b50a01948ad8e83e665cd7cdb3918994c07922dd18a1f7ff15ac8"}
Jul 28 16:02:24.816257 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.815826    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd" event={"ID":"46494030-ddac-40b4-9370-e89ae7b3102e","Type":"ContainerStarted","Data":"9940de1acabdfbe91da79852a95f959c095d4b5828e29e3ad8789e8eac6bb5db"}
Jul 28 16:02:24.819066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.818956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2hxs" event={"ID":"9da7f48a-38cf-4878-bb8d-53c58670b5d7","Type":"ContainerStarted","Data":"16182a6172e700686859be735b69548dabfa56935a7c7f24eecc11db99b5cce2"}
Jul 28 16:02:24.819066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.819027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2hxs" event={"ID":"9da7f48a-38cf-4878-bb8d-53c58670b5d7","Type":"ContainerStarted","Data":"cea24484bca7cc5d2a2f194dafc7f588fe9cb48c05c4d01bb70a69a99906649d"}
Jul 28 16:02:24.822774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.822734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh" event={"ID":"512d7939-27c8-435a-a509-d43fa1179b76","Type":"ContainerStarted","Data":"c281ea212c98fc6b6d480ea8c7d4092ba7a7985edf2b5a51cf09a057af272daf"}
Jul 28 16:02:24.842683 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:24.841989    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v2hxs" podStartSLOduration=65.694057403 podStartE2EDuration="1m8.841965296s" totalImagesPullingTime="3.147907893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:24.841289153 +0000 UTC m=+69.892093229" watchObservedRunningTime="2026-07-28 16:02:24.841965296 +0000 UTC m=+69.892769370"
Jul 28 16:02:25.233090 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.233046    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"]
Jul 28 16:02:25.238781 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.238217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.243961 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.243376    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 16:02:25.243961 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.243640    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 16:02:25.243961 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.243788    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 16:02:25.243961 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.243868    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 16:02:25.244675 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.244362    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 16:02:25.244675 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.244589    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tgxrg\""
Jul 28 16:02:25.244781 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.244752    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6jgjb5tjv8h34\""
Jul 28 16:02:25.257744 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.257719    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"]
Jul 28 16:02:25.327456 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9648d0-e141-4ded-b134-68e8a6a0d655-metrics-client-ca\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327480    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-grpc-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrz9x\" (UniqueName: \"kubernetes.io/projected/3b9648d0-e141-4ded-b134-68e8a6a0d655-kube-api-access-zrz9x\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327568    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327654 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327934 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.327934 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.327723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.428997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.428959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrz9x\" (UniqueName: \"kubernetes.io/projected/3b9648d0-e141-4ded-b134-68e8a6a0d655-kube-api-access-zrz9x\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429014    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429043    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429072    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429093    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429386 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429351    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429436 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9648d0-e141-4ded-b134-68e8a6a0d655-metrics-client-ca\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.429481 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.429454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-grpc-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.430266 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.430194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9648d0-e141-4ded-b134-68e8a6a0d655-metrics-client-ca\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.432407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.432378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.433606 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.433581    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.433962 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.433943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.434081 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.434057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.435072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.435044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.435173 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.435151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9648d0-e141-4ded-b134-68e8a6a0d655-secret-grpc-tls\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.440994 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.440970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrz9x\" (UniqueName: \"kubernetes.io/projected/3b9648d0-e141-4ded-b134-68e8a6a0d655-kube-api-access-zrz9x\") pod \"thanos-querier-5bb87b85b4-fdrxg\" (UID: \"3b9648d0-e141-4ded-b134-68e8a6a0d655\") " pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:25.552379 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:25.552297    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:27.878195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.878153    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"]
Jul 28 16:02:27.882512 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.882486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:27.885264 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.885208    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 16:02:27.885807 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.885775    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wzlkc\""
Jul 28 16:02:27.891328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.891292    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"]
Jul 28 16:02:27.950637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.950593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jnt\" (UniqueName: \"kubernetes.io/projected/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-kube-api-access-66jnt\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:27.950821 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:27.950779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.051587 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.051549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.051779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.051599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jnt\" (UniqueName: \"kubernetes.io/projected/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-kube-api-access-66jnt\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.051779 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:28.051744    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 16:02:28.051882 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:02:28.051843    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert podName:cdc9690c-b4ff-454b-b95c-fdadcde0d85e nodeName:}" failed. No retries permitted until 2026-07-28 16:02:28.551820971 +0000 UTC m=+73.602625040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-ttvjg" (UID: "cdc9690c-b4ff-454b-b95c-fdadcde0d85e") : secret "monitoring-plugin-cert" not found
Jul 28 16:02:28.063331 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.063297    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jnt\" (UniqueName: \"kubernetes.io/projected/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-kube-api-access-66jnt\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.556300 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.556256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.559176 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.559116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cdc9690c-b4ff-454b-b95c-fdadcde0d85e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ttvjg\" (UID: \"cdc9690c-b4ff-454b-b95c-fdadcde0d85e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.796804 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.796717    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:28.842836 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:28.842797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w746b" event={"ID":"4bfcea3d-07d4-435e-80a2-7b3787f35f04","Type":"ContainerStarted","Data":"0b427edb40e36fc978b48f934647a38ddf7fa1ef676a8385eb257bed04e30065"}
Jul 28 16:02:31.493517 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.493479    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:02:31.549850 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.549808    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:02:31.550033 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.549927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.552904 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.552881    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 16:02:31.553457 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.553434    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 16:02:31.553597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.553574    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 16:02:31.554140 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.554118    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 16:02:31.554140 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.554129    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9vttv\""
Jul 28 16:02:31.555027 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.555006    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 16:02:31.558474 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.558458    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 16:02:31.584814 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584860    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9x8\" (UniqueName: \"kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.584997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.585236 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.585033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686132 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686093    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9x8\" (UniqueName: \"kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686147    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686236    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.686570 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.686335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.687176 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.687140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.687302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.687175    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.687377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.687352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.687959 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.687932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.689419 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.689395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.689608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.689588    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.696578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.696555    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9x8\" (UniqueName: \"kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8\") pod \"console-58696dd668-x629z\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") " pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:31.860945 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:31.860911    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:36.838253 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.838221    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"]
Jul 28 16:02:36.880285 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.879654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"]
Jul 28 16:02:36.880285 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.879741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd" event={"ID":"46494030-ddac-40b4-9370-e89ae7b3102e","Type":"ContainerStarted","Data":"8baeb7bc328c3b6c1cb28e4d4eea70750dc2efeb19a3d11f10cd7a626283f5ce"}
Jul 28 16:02:36.880791 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.880767    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:02:36.881486 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.881452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j225x" event={"ID":"17c73738-c4bf-40fc-ad8c-7bac3ad2db11","Type":"ContainerStarted","Data":"83a34893a68fe3a10ecd7f6d3016fcf61ad604bd0f10a78f339cb766024f9156"}
Jul 28 16:02:36.882976 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.882955    2556 generic.go:356] "Generic (PLEG): container finished" podID="4bfcea3d-07d4-435e-80a2-7b3787f35f04" containerID="0b427edb40e36fc978b48f934647a38ddf7fa1ef676a8385eb257bed04e30065" exitCode=0
Jul 28 16:02:36.883073 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.882998    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w746b" event={"ID":"4bfcea3d-07d4-435e-80a2-7b3787f35f04","Type":"ContainerDied","Data":"0b427edb40e36fc978b48f934647a38ddf7fa1ef676a8385eb257bed04e30065"}
Jul 28 16:02:36.902303 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:36.902265    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc9690c_b4ff_454b_b95c_fdadcde0d85e.slice/crio-9a9e235cab00778b9e1bb637de7746a20a7708da817e439e215bac7be01d1969 WatchSource:0}: Error finding container 9a9e235cab00778b9e1bb637de7746a20a7708da817e439e215bac7be01d1969: Status 404 returned error can't find the container with id 9a9e235cab00778b9e1bb637de7746a20a7708da817e439e215bac7be01d1969
Jul 28 16:02:36.903957 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:36.903859    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ef66ae_7ec7_45a5_b888_ac2f97ed7064.slice/crio-68c511348c035ed4f191bd1cfdf1e28f2602252581068627f2b734546e78425a WatchSource:0}: Error finding container 68c511348c035ed4f191bd1cfdf1e28f2602252581068627f2b734546e78425a: Status 404 returned error can't find the container with id 68c511348c035ed4f191bd1cfdf1e28f2602252581068627f2b734546e78425a
Jul 28 16:02:36.930892 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.930853    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-p25nd" podStartSLOduration=1.79702856 podStartE2EDuration="13.93083645s" totalImagesPullingTime="12.13380789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:36.930228653 +0000 UTC m=+81.981032727" watchObservedRunningTime="2026-07-28 16:02:36.93083645 +0000 UTC m=+81.981640523"
Jul 28 16:02:36.958648 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:36.958570    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j225x" podStartSLOduration=64.926109151 podStartE2EDuration="1m20.95854865s" totalImagesPullingTime="16.032439499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:36.957277538 +0000 UTC m=+82.008081610" watchObservedRunningTime="2026-07-28 16:02:36.95854865 +0000 UTC m=+82.009352723"
Jul 28 16:02:37.893574 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.893528    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh" event={"ID":"512d7939-27c8-435a-a509-d43fa1179b76","Type":"ContainerStarted","Data":"14a42756d4204eff18bc079df4d280214fff6841fe0e8cd5453b0a185171bf60"}
Jul 28 16:02:37.894192 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.893614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh" event={"ID":"512d7939-27c8-435a-a509-d43fa1179b76","Type":"ContainerStarted","Data":"a97bd889112b7fa362b363179b3c5df6a015401aa53b1b9cc3fa0cdf043c901f"}
Jul 28 16:02:37.894192 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.893658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh" event={"ID":"512d7939-27c8-435a-a509-d43fa1179b76","Type":"ContainerStarted","Data":"331857dd4f5e7970d61cb1cbd4b0416486e5afb58d213f8496dea0dab8db9a79"}
Jul 28 16:02:37.896698 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.896664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"82f74d85b30e634eff850be239d4c199809e4f1e9bca83ccd8ff3c0c60dbd275"}
Jul 28 16:02:37.898500 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.898466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58696dd668-x629z" event={"ID":"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064","Type":"ContainerStarted","Data":"68c511348c035ed4f191bd1cfdf1e28f2602252581068627f2b734546e78425a"}
Jul 28 16:02:37.900891 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.900855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-v5vhn" event={"ID":"45254f28-7160-4e38-b5d9-ef3d1aad5181","Type":"ContainerStarted","Data":"7124e11d39aae987fa3d46a64b935f5a45df7b700f20be1508d62b0babe7f7d0"}
Jul 28 16:02:37.901030 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.900910    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:37.903897 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.903829    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bfrcj" event={"ID":"a4e33799-9ac4-4628-a0fb-170ceda351bb","Type":"ContainerStarted","Data":"f3f90502d4a7dacfb20ec1c87aae9cb5b202d908300980bdea2e51194362ac64"}
Jul 28 16:02:37.904031 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.903970    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:02:37.907294 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.906598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w746b" event={"ID":"4bfcea3d-07d4-435e-80a2-7b3787f35f04","Type":"ContainerStarted","Data":"319ee1d8630831e0982299a49858132aaba7bf8c0f950b6a8e095796c5c6443a"}
Jul 28 16:02:37.907294 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.906672    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w746b" event={"ID":"4bfcea3d-07d4-435e-80a2-7b3787f35f04","Type":"ContainerStarted","Data":"777df5694b5477c378e806c35e9357e5ceff6616caabb990d795ff24acef61ba"}
Jul 28 16:02:37.911004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.910973    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg" event={"ID":"cdc9690c-b4ff-454b-b95c-fdadcde0d85e","Type":"ContainerStarted","Data":"9a9e235cab00778b9e1bb637de7746a20a7708da817e439e215bac7be01d1969"}
Jul 28 16:02:37.914199 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.914148    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-v5vhn"
Jul 28 16:02:37.937161 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.937075    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-jsffh" podStartSLOduration=1.966358478 podStartE2EDuration="14.937054286s" totalImagesPullingTime="12.970695808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:37.935059601 +0000 UTC m=+82.985863690" watchObservedRunningTime="2026-07-28 16:02:37.937054286 +0000 UTC m=+82.987858360"
Jul 28 16:02:37.977081 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:37.977010    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w746b" podStartSLOduration=13.189472952 podStartE2EDuration="14.976987787s" totalImagesPullingTime="1.680771171s" totalInitContainerRuntime="106.743664ms" isStatefulPod=true podCreationTimestamp="2026-07-28 16:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:37.976012329 +0000 UTC m=+83.026816604" watchObservedRunningTime="2026-07-28 16:02:37.976987787 +0000 UTC m=+83.027791861"
Jul 28 16:02:38.063384 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:38.063317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bfrcj" podStartSLOduration=66.049834106 podStartE2EDuration="1m22.063292832s" totalImagesPullingTime="16.013458726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.01376767 +0000 UTC m=+83.064571744" watchObservedRunningTime="2026-07-28 16:02:38.063292832 +0000 UTC m=+83.114096906"
Jul 28 16:02:39.776559 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.776487    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-v5vhn" podStartSLOduration=5.646978295 podStartE2EDuration="27.776461195s" totalImagesPullingTime="22.1294829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:38.064567762 +0000 UTC m=+83.115371835" watchObservedRunningTime="2026-07-28 16:02:39.776461195 +0000 UTC m=+84.827265272"
Jul 28 16:02:39.777522 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.777487    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:02:39.841090 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.841058    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:02:39.878552 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.878515    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:02:39.878743 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.878587    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglxz\" (UniqueName: \"kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.990758 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:39.991607 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:39.990607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.092773 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092685    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.092773 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.093311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.093311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.093311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.093311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglxz\" (UniqueName: \"kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.093311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.092960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.095719 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.094355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.095719 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.095205    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.095719 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.095673    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.096322 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.096292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.097410 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.097313    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.098220 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.097601    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.107792 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.107758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglxz\" (UniqueName: \"kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz\") pod \"console-9bcbddf69-bjsgp\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") " pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:40.190802 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:40.190447    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:42.523579 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.523546    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:02:42.646731 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:02:42.646690    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84dd0de_4f7b_4593_bce7_976c5a73722d.slice/crio-126017ee69bc28a23c9203fd64e06b9d710b01c2f2f3229ea5357d3af27d95df WatchSource:0}: Error finding container 126017ee69bc28a23c9203fd64e06b9d710b01c2f2f3229ea5357d3af27d95df: Status 404 returned error can't find the container with id 126017ee69bc28a23c9203fd64e06b9d710b01c2f2f3229ea5357d3af27d95df
Jul 28 16:02:42.937757 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.937659    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg" event={"ID":"cdc9690c-b4ff-454b-b95c-fdadcde0d85e","Type":"ContainerStarted","Data":"024ebb8d75d7c61e1769808c884cb1fafa5b17826cf34685a8297d8807a99167"}
Jul 28 16:02:42.937998 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.937973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:42.939987 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.939953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bcbddf69-bjsgp" event={"ID":"c84dd0de-4f7b-4593-bce7-976c5a73722d","Type":"ContainerStarted","Data":"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"}
Jul 28 16:02:42.940135 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.940004    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bcbddf69-bjsgp" event={"ID":"c84dd0de-4f7b-4593-bce7-976c5a73722d","Type":"ContainerStarted","Data":"126017ee69bc28a23c9203fd64e06b9d710b01c2f2f3229ea5357d3af27d95df"}
Jul 28 16:02:42.942806 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.942775    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"f22247603094a5495ca82c68c4636cc681fcc9b5540c1fd2c1d2e6b90d67e35d"}
Jul 28 16:02:42.942930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.942849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"29c2e13f2113077904ec5ec0135caf173122815d38f84371121fa85946aeefa6"}
Jul 28 16:02:42.942930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.942893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"8a980ee53c1d2ec9539238d7cf2561a8ccf3d0192080f1dba92a375d76b6bcb6"}
Jul 28 16:02:42.944092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.944063    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg"
Jul 28 16:02:42.944700 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.944668    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58696dd668-x629z" event={"ID":"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064","Type":"ContainerStarted","Data":"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"}
Jul 28 16:02:42.987404 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:42.987340    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ttvjg" podStartSLOduration=10.2472428 podStartE2EDuration="15.987290249s" totalImagesPullingTime="5.740047449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:42.961773872 +0000 UTC m=+88.012577946" watchObservedRunningTime="2026-07-28 16:02:42.987290249 +0000 UTC m=+88.038094323"
Jul 28 16:02:43.023534 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:43.023483    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58696dd668-x629z" podStartSLOduration=6.61784482 podStartE2EDuration="12.023464495s" totalImagesPullingTime="5.405619675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:43.021999477 +0000 UTC m=+88.072803576" watchObservedRunningTime="2026-07-28 16:02:43.023464495 +0000 UTC m=+88.074268568"
Jul 28 16:02:43.049307 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:43.049243    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9bcbddf69-bjsgp" podStartSLOduration=4.0492198 podStartE2EDuration="4.0492198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:43.047728553 +0000 UTC m=+88.098532631" watchObservedRunningTime="2026-07-28 16:02:43.0492198 +0000 UTC m=+88.100023875"
Jul 28 16:02:44.958359 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:44.958275    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"e8137a97a230397a6c9828ac114707fbfbfb8ffad6b0a9d088f8ea63ede2bd2f"}
Jul 28 16:02:44.958359 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:44.958355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"0299a1c30b8efb51c8db29c31d9fb8df01e3647cda62e28b08bf43add4c43afb"}
Jul 28 16:02:44.958956 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:44.958399    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" event={"ID":"3b9648d0-e141-4ded-b134-68e8a6a0d655","Type":"ContainerStarted","Data":"9c5395caeb76c8d987dc46c5d4fbf675bb5ba6477df2a0b5304433fdf57c76f6"}
Jul 28 16:02:44.991792 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:44.991734    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg" podStartSLOduration=13.096243099 podStartE2EDuration="19.991716415s" totalImagesPullingTime="6.895473316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:02:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:02:44.98923343 +0000 UTC m=+90.040037518" watchObservedRunningTime="2026-07-28 16:02:44.991716415 +0000 UTC m=+90.042520501"
Jul 28 16:02:45.962607 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:45.962568    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:02:50.192016 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:50.191972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:50.192577 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:50.192057    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:50.197224 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:50.197200    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:50.990010 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:50.989982    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:02:51.861566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:51.861528    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:02:51.971812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:02:51.971785    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5bb87b85b4-fdrxg"
Jul 28 16:03:07.967830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:07.967768    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58696dd668-x629z" podUID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" containerName="console" containerID="cri-o://1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa" gracePeriod=15
Jul 28 16:03:08.206967 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.206943    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:03:08.237726 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237614    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.237726 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237692    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237746    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237776    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237801    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237838    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9x8\" (UniqueName: \"kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.237940    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert\") pod \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\" (UID: \"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064\") "
Jul 28 16:03:08.238295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.238229    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config" (OuterVolumeSpecName: "console-config") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:08.238427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.238249    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:08.238427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.238245    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:08.238637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.238578    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:03:08.241204 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.241166    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8" (OuterVolumeSpecName: "kube-api-access-rz9x8") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "kube-api-access-rz9x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:03:08.241328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.241216    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:03:08.241393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.241341    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" (UID: "f4ef66ae-7ec7-45a5-b888-ac2f97ed7064"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:03:08.339483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339442    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-oauth-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339479    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-oauth-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339487    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9x8\" (UniqueName: \"kubernetes.io/projected/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-kube-api-access-rz9x8\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339766 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339497    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339766 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339506    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-trusted-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339766 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339515    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-service-ca\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.339766 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.339523    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064-console-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:03:08.921993 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:08.921963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bfrcj"
Jul 28 16:03:09.054827 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.054793    2556 generic.go:356] "Generic (PLEG): container finished" podID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" containerID="1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa" exitCode=0
Jul 28 16:03:09.055212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.054848    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58696dd668-x629z"
Jul 28 16:03:09.055212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.054888    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58696dd668-x629z" event={"ID":"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064","Type":"ContainerDied","Data":"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"}
Jul 28 16:03:09.055212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.054945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58696dd668-x629z" event={"ID":"f4ef66ae-7ec7-45a5-b888-ac2f97ed7064","Type":"ContainerDied","Data":"68c511348c035ed4f191bd1cfdf1e28f2602252581068627f2b734546e78425a"}
Jul 28 16:03:09.055212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.054971    2556 scope.go:117] "RemoveContainer" containerID="1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"
Jul 28 16:03:09.066226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.066202    2556 scope.go:117] "RemoveContainer" containerID="1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"
Jul 28 16:03:09.066543 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:03:09.066522    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa\": container with ID starting with 1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa not found: ID does not exist" containerID="1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"
Jul 28 16:03:09.066603 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.066557    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa"} err="failed to get container status \"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa\": rpc error: code = NotFound desc = could not find container \"1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa\": container with ID starting with 1ff77964ea8d4dac85798a6d530e87e525da1d711e32f272f9ca20e2c3c1b3fa not found: ID does not exist"
Jul 28 16:03:09.078074 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.078037    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:03:09.082417 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.082388    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58696dd668-x629z"]
Jul 28 16:03:09.461994 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:09.461958    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" path="/var/lib/kubelet/pods/f4ef66ae-7ec7-45a5-b888-ac2f97ed7064/volumes"
Jul 28 16:03:45.185174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.185135    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:03:45.185653 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.185411    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" containerName="console"
Jul 28 16:03:45.185653 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.185424    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" containerName="console"
Jul 28 16:03:45.185653 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.185489    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4ef66ae-7ec7-45a5-b888-ac2f97ed7064" containerName="console"
Jul 28 16:03:45.188849 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.188826    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.191573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.191549    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 16:03:45.191742 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.191584    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 16:03:45.193178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.193157    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 16:03:45.193720 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.193689    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 16:03:45.194384 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.194366    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 16:03:45.194868 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.194849    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 16:03:45.195407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.195379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 16:03:45.195960 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.195946    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hq6rk\""
Jul 28 16:03:45.196231 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.196215    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 16:03:45.198748 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.198730    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 16:03:45.214769 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.214732    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:03:45.331387 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331387 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-out\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-web-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcn5\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-kube-api-access-mrcn5\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.331832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.331814    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432257 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432257 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-web-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432424    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcn5\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-kube-api-access-mrcn5\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432455    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432679    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432756    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432911 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432814    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.432911 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432872    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.433014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.433014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.433014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.432991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-out\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.433906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.433513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.433906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.433595    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.435918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.435981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-web-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.435997    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.436031    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-out\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436167 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.436078    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436439 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.436320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436439 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.436420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.436546 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.436525    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.437964 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.437948    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.441691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.441674    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcn5\" (UniqueName: \"kubernetes.io/projected/b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8-kube-api-access-mrcn5\") pod \"alertmanager-main-0\" (UID: \"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.500386 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.500334    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 16:03:45.646364 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:45.646329    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 16:03:45.650209 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:03:45.650178    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb724ab0f_3647_4b00_a4fe_f8a1d0b0a6b8.slice/crio-3f8fdb3ac0660d62f322b1257ed1e3055727b9b1c858964725d523d12890184d WatchSource:0}: Error finding container 3f8fdb3ac0660d62f322b1257ed1e3055727b9b1c858964725d523d12890184d: Status 404 returned error can't find the container with id 3f8fdb3ac0660d62f322b1257ed1e3055727b9b1c858964725d523d12890184d
Jul 28 16:03:46.187007 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:46.186973    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"3f8fdb3ac0660d62f322b1257ed1e3055727b9b1c858964725d523d12890184d"}
Jul 28 16:03:47.191889 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:47.191849    2556 generic.go:356] "Generic (PLEG): container finished" podID="b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8" containerID="65e4058612b3711354dadadeeabff8121aa8677d99122c923e157a9d66e9dcf8" exitCode=0
Jul 28 16:03:47.192293 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:47.191895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerDied","Data":"65e4058612b3711354dadadeeabff8121aa8677d99122c923e157a9d66e9dcf8"}
Jul 28 16:03:49.203563 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:49.203532    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"7077d2b9ae266c0fd8267618745d705fa7b33facae9cfd1f83af948736bd0665"}
Jul 28 16:03:50.210592 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.210555    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"10be352bcf68baa59a4e8ef1d79241524c6e5cade9d54586b3beb2ff105ef63a"}
Jul 28 16:03:50.210980 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.210639    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"db335677b9dacd745a84b6f9001df0c0ed3cae73973c230ea0c8b2efc59c7db4"}
Jul 28 16:03:50.210980 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.210666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"b554020e5ba185c2ddc985fd83368429b99d45b9f4c1d9a236380d7e2a6283a9"}
Jul 28 16:03:50.210980 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.210692    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"225d92871fc73a78f16d85a3b395d3f0ae47d06f4daa18fcf78a189be6287898"}
Jul 28 16:03:50.210980 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.210721    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8","Type":"ContainerStarted","Data":"0beabd93fa0ef2222bee765d1c628f7cc263822bb26c77bc1f74e6e7c5e75cd5"}
Jul 28 16:03:50.240571 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:03:50.240513    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.002767122 podStartE2EDuration="5.240494571s" totalImagesPullingTime="3.220586256s" totalInitContainerRuntime="17.141193ms" isStatefulPod=false podCreationTimestamp="2026-07-28 16:03:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:03:50.238365809 +0000 UTC m=+155.289169884" watchObservedRunningTime="2026-07-28 16:03:50.240494571 +0000 UTC m=+155.291298644"
Jul 28 16:04:08.042174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.042143    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:04:08.044335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.044318    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.064616 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.064573    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:04:08.118765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ktsf\" (UniqueName: \"kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.118997 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.118917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220006 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.219970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ktsf\" (UniqueName: \"kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220006 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220030    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220077    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220139    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.220927 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.221073 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.221073 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.220961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.221303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.221268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.222637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.222607    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.222996 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.222979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.229822 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.229792    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ktsf\" (UniqueName: \"kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf\") pod \"console-9d645d9c4-dmt4n\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") " pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.353977 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.353938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:08.492611 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:08.492576    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:04:08.495497 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:04:08.495470    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99ef047_50af_4b22_bbdd_5d43277e6b53.slice/crio-cb274ca22f9e9a05c7c4ac554fd0a79713155e4f4d899800667b1e6323b639e2 WatchSource:0}: Error finding container cb274ca22f9e9a05c7c4ac554fd0a79713155e4f4d899800667b1e6323b639e2: Status 404 returned error can't find the container with id cb274ca22f9e9a05c7c4ac554fd0a79713155e4f4d899800667b1e6323b639e2
Jul 28 16:04:09.286515 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:09.286477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d645d9c4-dmt4n" event={"ID":"a99ef047-50af-4b22-bbdd-5d43277e6b53","Type":"ContainerStarted","Data":"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"}
Jul 28 16:04:09.286963 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:09.286537    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d645d9c4-dmt4n" event={"ID":"a99ef047-50af-4b22-bbdd-5d43277e6b53","Type":"ContainerStarted","Data":"cb274ca22f9e9a05c7c4ac554fd0a79713155e4f4d899800667b1e6323b639e2"}
Jul 28 16:04:09.306192 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:09.306124    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d645d9c4-dmt4n" podStartSLOduration=1.3061021099999999 podStartE2EDuration="1.30610211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:04:09.305124715 +0000 UTC m=+174.355928798" watchObservedRunningTime="2026-07-28 16:04:09.30610211 +0000 UTC m=+174.356906184"
Jul 28 16:04:18.354708 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:18.354670    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:18.355388 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:18.354737    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:18.359452 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:18.359427    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:19.329354 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:19.329324    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:04:19.395096 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:19.395064    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:04:44.414908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.414849    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9bcbddf69-bjsgp" podUID="c84dd0de-4f7b-4593-bce7-976c5a73722d" containerName="console" containerID="cri-o://d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a" gracePeriod=15
Jul 28 16:04:44.663220 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.663196    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:04:44.731361 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731270    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731361 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731316    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qglxz\" (UniqueName: \"kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731361 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731345    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731522    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731567    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731600    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731671    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config\") pod \"c84dd0de-4f7b-4593-bce7-976c5a73722d\" (UID: \"c84dd0de-4f7b-4593-bce7-976c5a73722d\") "
Jul 28 16:04:44.731824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731714    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:44.731951 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731927    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:44.732018 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731934    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-oauth-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.732018 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.731963    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca" (OuterVolumeSpecName: "service-ca") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:44.732184 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.732159    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config" (OuterVolumeSpecName: "console-config") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:04:44.733706 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.733678    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:44.733854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.733834    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz" (OuterVolumeSpecName: "kube-api-access-qglxz") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "kube-api-access-qglxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:04:44.733913 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.733863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c84dd0de-4f7b-4593-bce7-976c5a73722d" (UID: "c84dd0de-4f7b-4593-bce7-976c5a73722d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:04:44.832691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832646    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglxz\" (UniqueName: \"kubernetes.io/projected/c84dd0de-4f7b-4593-bce7-976c5a73722d-kube-api-access-qglxz\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.832691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832684    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-oauth-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.832691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832696    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.832930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832709    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-service-ca\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.832930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832721    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-trusted-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:44.832930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:44.832733    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c84dd0de-4f7b-4593-bce7-976c5a73722d-console-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:04:45.426365 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.426327    2556 generic.go:356] "Generic (PLEG): container finished" podID="c84dd0de-4f7b-4593-bce7-976c5a73722d" containerID="d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a" exitCode=0
Jul 28 16:04:45.426812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.426375    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bcbddf69-bjsgp"
Jul 28 16:04:45.426812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.426398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bcbddf69-bjsgp" event={"ID":"c84dd0de-4f7b-4593-bce7-976c5a73722d","Type":"ContainerDied","Data":"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"}
Jul 28 16:04:45.426812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.426453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bcbddf69-bjsgp" event={"ID":"c84dd0de-4f7b-4593-bce7-976c5a73722d","Type":"ContainerDied","Data":"126017ee69bc28a23c9203fd64e06b9d710b01c2f2f3229ea5357d3af27d95df"}
Jul 28 16:04:45.426812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.426474    2556 scope.go:117] "RemoveContainer" containerID="d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"
Jul 28 16:04:45.439151 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.439128    2556 scope.go:117] "RemoveContainer" containerID="d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"
Jul 28 16:04:45.439484 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:04:45.439465    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a\": container with ID starting with d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a not found: ID does not exist" containerID="d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"
Jul 28 16:04:45.439534 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.439492    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a"} err="failed to get container status \"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a\": rpc error: code = NotFound desc = could not find container \"d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a\": container with ID starting with d22c15f4891b68876b043027dffaf87f8b20caff0afb778a9521a1a21563969a not found: ID does not exist"
Jul 28 16:04:45.450332 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.450302    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:04:45.453419 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.453392    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9bcbddf69-bjsgp"]
Jul 28 16:04:45.464031 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:04:45.463999    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84dd0de-4f7b-4593-bce7-976c5a73722d" path="/var/lib/kubelet/pods/c84dd0de-4f7b-4593-bce7-976c5a73722d/volumes"
Jul 28 16:05:09.221672 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.221616    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"]
Jul 28 16:05:09.222146 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.221962    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84dd0de-4f7b-4593-bce7-976c5a73722d" containerName="console"
Jul 28 16:05:09.222146 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.221973    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84dd0de-4f7b-4593-bce7-976c5a73722d" containerName="console"
Jul 28 16:05:09.222146 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.222025    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84dd0de-4f7b-4593-bce7-976c5a73722d" containerName="console"
Jul 28 16:05:09.223892 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.223874    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.227718 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.227689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wsm\" (UniqueName: \"kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.227860 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.227818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.227918 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.227858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.229948 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.229925    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 16:05:09.230083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.230000    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 16:05:09.230883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.230868    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5w5kw\""
Jul 28 16:05:09.238539 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.238506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"]
Jul 28 16:05:09.329129 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.329094    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.329301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.329138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.329301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.329175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wsm\" (UniqueName: \"kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.329556 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.329531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.329592 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.329542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.338174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.338148    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wsm\" (UniqueName: \"kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.533193 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.533103    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:09.672244 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:09.672202    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"]
Jul 28 16:05:09.675261 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:05:09.675219    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdfe07a_7428_452f_98e6_61f89e802386.slice/crio-946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea WatchSource:0}: Error finding container 946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea: Status 404 returned error can't find the container with id 946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea
Jul 28 16:05:10.526292 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:10.526255    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerStarted","Data":"946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea"}
Jul 28 16:05:17.560122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:17.560030    2556 generic.go:356] "Generic (PLEG): container finished" podID="7cdfe07a-7428-452f-98e6-61f89e802386" containerID="0e1bad6c4d7ae51ebb955ce088a54f94fe65a3c47f313c87cbd3d30c825a2eab" exitCode=0
Jul 28 16:05:17.560122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:17.560102    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerDied","Data":"0e1bad6c4d7ae51ebb955ce088a54f94fe65a3c47f313c87cbd3d30c825a2eab"}
Jul 28 16:05:20.576022 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:20.575930    2556 generic.go:356] "Generic (PLEG): container finished" podID="7cdfe07a-7428-452f-98e6-61f89e802386" containerID="26214cf2c97df8d962f57ac5f7900ee76a97989370c5f534fb0d688046222461" exitCode=0
Jul 28 16:05:20.576022 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:20.575986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerDied","Data":"26214cf2c97df8d962f57ac5f7900ee76a97989370c5f534fb0d688046222461"}
Jul 28 16:05:31.622775 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:31.622736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerStarted","Data":"7397120e9476183e6357fd07bd795a1c7945f7b3e55471bda2373f69e85a1f16"}
Jul 28 16:05:31.644504 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:31.644434    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" podStartSLOduration=1.62560507 podStartE2EDuration="22.644413374s" totalImagesPullingTime="21.018808304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:31.644172355 +0000 UTC m=+256.694976432" watchObservedRunningTime="2026-07-28 16:05:31.644413374 +0000 UTC m=+256.695217449"
Jul 28 16:05:32.628832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:32.628794    2556 generic.go:356] "Generic (PLEG): container finished" podID="7cdfe07a-7428-452f-98e6-61f89e802386" containerID="7397120e9476183e6357fd07bd795a1c7945f7b3e55471bda2373f69e85a1f16" exitCode=0
Jul 28 16:05:32.629263 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:32.628874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerDied","Data":"7397120e9476183e6357fd07bd795a1c7945f7b3e55471bda2373f69e85a1f16"}
Jul 28 16:05:33.772858 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.772823    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:33.843087 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.843050    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85wsm\" (UniqueName: \"kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm\") pod \"7cdfe07a-7428-452f-98e6-61f89e802386\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") "
Jul 28 16:05:33.843303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.843108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle\") pod \"7cdfe07a-7428-452f-98e6-61f89e802386\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") "
Jul 28 16:05:33.843303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.843144    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util\") pod \"7cdfe07a-7428-452f-98e6-61f89e802386\" (UID: \"7cdfe07a-7428-452f-98e6-61f89e802386\") "
Jul 28 16:05:33.843779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.843753    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle" (OuterVolumeSpecName: "bundle") pod "7cdfe07a-7428-452f-98e6-61f89e802386" (UID: "7cdfe07a-7428-452f-98e6-61f89e802386"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:33.845515 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.845491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm" (OuterVolumeSpecName: "kube-api-access-85wsm") pod "7cdfe07a-7428-452f-98e6-61f89e802386" (UID: "7cdfe07a-7428-452f-98e6-61f89e802386"). InnerVolumeSpecName "kube-api-access-85wsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:05:33.847178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.847157    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util" (OuterVolumeSpecName: "util") pod "7cdfe07a-7428-452f-98e6-61f89e802386" (UID: "7cdfe07a-7428-452f-98e6-61f89e802386"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 16:05:33.944198 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.944112    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wsm\" (UniqueName: \"kubernetes.io/projected/7cdfe07a-7428-452f-98e6-61f89e802386-kube-api-access-85wsm\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:05:33.944198 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.944146    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:05:33.944198 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:33.944154    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cdfe07a-7428-452f-98e6-61f89e802386-util\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:05:34.639603 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:34.639578    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs"
Jul 28 16:05:34.639603 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:34.639573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5pkvbs" event={"ID":"7cdfe07a-7428-452f-98e6-61f89e802386","Type":"ContainerDied","Data":"946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea"}
Jul 28 16:05:34.639603 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:34.639682    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="946bfaf9da46dffdc3461dc33f10a279a3bf4d034a817e79d39d60605af997ea"
Jul 28 16:05:37.014031 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.013952    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"]
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014279    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="util"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014289    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="util"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014296    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="extract"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014301    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="extract"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014320    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="pull"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014325    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="pull"
Jul 28 16:05:37.014455 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.014368    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cdfe07a-7428-452f-98e6-61f89e802386" containerName="extract"
Jul 28 16:05:37.016291 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.016270    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.018853 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.018827    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 16:05:37.018987 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.018852    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qljdr\""
Jul 28 16:05:37.018987 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.018827    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 16:05:37.018987 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.018835    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 16:05:37.033953 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.033925    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"]
Jul 28 16:05:37.170079 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.170039    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/13bb3d78-976b-40b7-b885-5a3c8d1c0744-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.170395 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.170088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtl4b\" (UniqueName: \"kubernetes.io/projected/13bb3d78-976b-40b7-b885-5a3c8d1c0744-kube-api-access-xtl4b\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.270929 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.270829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/13bb3d78-976b-40b7-b885-5a3c8d1c0744-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.270929 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.270874    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtl4b\" (UniqueName: \"kubernetes.io/projected/13bb3d78-976b-40b7-b885-5a3c8d1c0744-kube-api-access-xtl4b\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.273439 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.273411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/13bb3d78-976b-40b7-b885-5a3c8d1c0744-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.280079 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.280044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtl4b\" (UniqueName: \"kubernetes.io/projected/13bb3d78-976b-40b7-b885-5a3c8d1c0744-kube-api-access-xtl4b\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-gdssp\" (UID: \"13bb3d78-976b-40b7-b885-5a3c8d1c0744\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.327092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.327058    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:37.469355 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.469286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"]
Jul 28 16:05:37.473667 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:05:37.473640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bb3d78_976b_40b7_b885_5a3c8d1c0744.slice/crio-46ed7c9222de3a90bffbe61ec87763bc78518cb6557e34c9fbc3f492cb0fb25e WatchSource:0}: Error finding container 46ed7c9222de3a90bffbe61ec87763bc78518cb6557e34c9fbc3f492cb0fb25e: Status 404 returned error can't find the container with id 46ed7c9222de3a90bffbe61ec87763bc78518cb6557e34c9fbc3f492cb0fb25e
Jul 28 16:05:37.654024 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:37.653987    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp" event={"ID":"13bb3d78-976b-40b7-b885-5a3c8d1c0744","Type":"ContainerStarted","Data":"46ed7c9222de3a90bffbe61ec87763bc78518cb6557e34c9fbc3f492cb0fb25e"}
Jul 28 16:05:40.671311 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:40.671271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp" event={"ID":"13bb3d78-976b-40b7-b885-5a3c8d1c0744","Type":"ContainerStarted","Data":"d69c69a4526f369f6fd88c4684c24c7479b4f7bb24bbeba8216139f829f91c9a"}
Jul 28 16:05:40.671712 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:40.671480    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:05:40.695467 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:40.695412    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp" podStartSLOduration=1.580621656 podStartE2EDuration="4.695396504s" totalImagesPullingTime="3.114774848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:40.693867011 +0000 UTC m=+265.744671084" watchObservedRunningTime="2026-07-28 16:05:40.695396504 +0000 UTC m=+265.746200577"
Jul 28 16:05:41.462994 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.462965    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-wc8z6"]
Jul 28 16:05:41.465067 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.465052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.468091 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.468069    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 16:05:41.468214 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.468099    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 16:05:41.468214 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.468196    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-bsgkn\""
Jul 28 16:05:41.476835 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.476809    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wc8z6"]
Jul 28 16:05:41.605011 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.604970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.605205 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.605048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76br8\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-kube-api-access-76br8\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.605205 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.605159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/19e200e6-ec2a-4b19-b719-59670ec1e5c1-cabundle0\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.706561 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.706525    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/19e200e6-ec2a-4b19-b719-59670ec1e5c1-cabundle0\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.706580    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.706607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76br8\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-kube-api-access-76br8\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:41.706732    2556 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:41.706755    2556 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:41.706765    2556 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:41.706780    2556 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-wc8z6: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:05:41.707079 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:41.706840    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates podName:19e200e6-ec2a-4b19-b719-59670ec1e5c1 nodeName:}" failed. No retries permitted until 2026-07-28 16:05:42.206821313 +0000 UTC m=+267.257625370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates") pod "keda-operator-77f648d694-wc8z6" (UID: "19e200e6-ec2a-4b19-b719-59670ec1e5c1") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 16:05:41.707468 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.707393    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/19e200e6-ec2a-4b19-b719-59670ec1e5c1-cabundle0\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:41.716403 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:41.716331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76br8\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-kube-api-access-76br8\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:42.094952 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.094916    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"]
Jul 28 16:05:42.097415 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.097399    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.099922 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.099900    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 16:05:42.106728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.106697    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"]
Jul 28 16:05:42.211442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.211391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.211685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.211466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wnv\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-kube-api-access-z6wnv\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.211685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.211503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:42.211685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.211605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.211685 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.211665    2556 secret.go:281] references non-existent secret key: ca.crt
Jul 28 16:05:42.211685 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.211684    2556 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 16:05:42.211888 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.211694    2556 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-wc8z6: references non-existent secret key: ca.crt
Jul 28 16:05:42.211888 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.211795    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates podName:19e200e6-ec2a-4b19-b719-59670ec1e5c1 nodeName:}" failed. No retries permitted until 2026-07-28 16:05:43.211776877 +0000 UTC m=+268.262580928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates") pod "keda-operator-77f648d694-wc8z6" (UID: "19e200e6-ec2a-4b19-b719-59670ec1e5c1") : references non-existent secret key: ca.crt
Jul 28 16:05:42.312287 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.312251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wnv\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-kube-api-access-z6wnv\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.312498 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.312315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.312498 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.312360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.312498 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.312473    2556 secret.go:281] references non-existent secret key: tls.crt
Jul 28 16:05:42.312498 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.312491    2556 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 16:05:42.312737 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.312513    2556 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr: references non-existent secret key: tls.crt
Jul 28 16:05:42.312737 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.312571    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates podName:1f07b048-4cbb-4722-ba73-4cb6c353d1a2 nodeName:}" failed. No retries permitted until 2026-07-28 16:05:42.812548922 +0000 UTC m=+267.863352998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates") pod "keda-metrics-apiserver-55557df8c-gbjfr" (UID: "1f07b048-4cbb-4722-ba73-4cb6c353d1a2") : references non-existent secret key: tls.crt
Jul 28 16:05:42.312937 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.312918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.322226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.322194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wnv\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-kube-api-access-z6wnv\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.702883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.702846    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-nj8w8"]
Jul 28 16:05:42.705443 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.705427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:42.708175 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.708146    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 16:05:42.716298 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.716272    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-nj8w8"]
Jul 28 16:05:42.816598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.816564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.816811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.816641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:42.816811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.816683    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdqh\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-kube-api-access-vcdqh\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:42.819635 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.819597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f07b048-4cbb-4722-ba73-4cb6c353d1a2-certificates\") pod \"keda-metrics-apiserver-55557df8c-gbjfr\" (UID: \"1f07b048-4cbb-4722-ba73-4cb6c353d1a2\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:42.917437 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.917398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:42.917644 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.917449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdqh\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-kube-api-access-vcdqh\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:42.917644 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.917570    2556 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 16:05:42.917644 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.917605    2556 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-nj8w8: secret "keda-admission-webhooks-certs" not found
Jul 28 16:05:42.917808 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:05:42.917682    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates podName:1f1a7ae2-8ee8-4891-95f3-71987273301e nodeName:}" failed. No retries permitted until 2026-07-28 16:05:43.41766491 +0000 UTC m=+268.468468963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates") pod "keda-admission-56f9798bb4-nj8w8" (UID: "1f1a7ae2-8ee8-4891-95f3-71987273301e") : secret "keda-admission-webhooks-certs" not found
Jul 28 16:05:42.928528 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:42.928497    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdqh\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-kube-api-access-vcdqh\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:43.008169 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.008067    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:43.170839 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.170799    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"]
Jul 28 16:05:43.175577 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:05:43.175544    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f07b048_4cbb_4722_ba73_4cb6c353d1a2.slice/crio-31cd47c40418b47d43eb90610ffb7148200029b6d6a62e3bb3dfdc7ffa55a420 WatchSource:0}: Error finding container 31cd47c40418b47d43eb90610ffb7148200029b6d6a62e3bb3dfdc7ffa55a420: Status 404 returned error can't find the container with id 31cd47c40418b47d43eb90610ffb7148200029b6d6a62e3bb3dfdc7ffa55a420
Jul 28 16:05:43.220003 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.219965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:43.223191 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.223160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/19e200e6-ec2a-4b19-b719-59670ec1e5c1-certificates\") pod \"keda-operator-77f648d694-wc8z6\" (UID: \"19e200e6-ec2a-4b19-b719-59670ec1e5c1\") " pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:43.275977 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.275883    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:43.422042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.422005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:43.424791 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.424759    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1f1a7ae2-8ee8-4891-95f3-71987273301e-certificates\") pod \"keda-admission-56f9798bb4-nj8w8\" (UID: \"1f1a7ae2-8ee8-4891-95f3-71987273301e\") " pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:43.425833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.425810    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wc8z6"]
Jul 28 16:05:43.429116 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:05:43.429092    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e200e6_ec2a_4b19_b719_59670ec1e5c1.slice/crio-d9f037bf9119e316147e3334e668f98725b4995f280405655ab1e07d78eb2a1d WatchSource:0}: Error finding container d9f037bf9119e316147e3334e668f98725b4995f280405655ab1e07d78eb2a1d: Status 404 returned error can't find the container with id d9f037bf9119e316147e3334e668f98725b4995f280405655ab1e07d78eb2a1d
Jul 28 16:05:43.617318 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.617283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:43.685811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.685777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wc8z6" event={"ID":"19e200e6-ec2a-4b19-b719-59670ec1e5c1","Type":"ContainerStarted","Data":"d9f037bf9119e316147e3334e668f98725b4995f280405655ab1e07d78eb2a1d"}
Jul 28 16:05:43.687133 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.687096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr" event={"ID":"1f07b048-4cbb-4722-ba73-4cb6c353d1a2","Type":"ContainerStarted","Data":"31cd47c40418b47d43eb90610ffb7148200029b6d6a62e3bb3dfdc7ffa55a420"}
Jul 28 16:05:43.755978 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:43.755949    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-nj8w8"]
Jul 28 16:05:43.758771 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:05:43.758732    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1a7ae2_8ee8_4891_95f3_71987273301e.slice/crio-18d7099708608ca6fdaa50754758c891c75ac27fb9e388ee6710c897ca8d6432 WatchSource:0}: Error finding container 18d7099708608ca6fdaa50754758c891c75ac27fb9e388ee6710c897ca8d6432: Status 404 returned error can't find the container with id 18d7099708608ca6fdaa50754758c891c75ac27fb9e388ee6710c897ca8d6432
Jul 28 16:05:44.694703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:44.694663    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8" event={"ID":"1f1a7ae2-8ee8-4891-95f3-71987273301e","Type":"ContainerStarted","Data":"18d7099708608ca6fdaa50754758c891c75ac27fb9e388ee6710c897ca8d6432"}
Jul 28 16:05:47.712992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.712890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wc8z6" event={"ID":"19e200e6-ec2a-4b19-b719-59670ec1e5c1","Type":"ContainerStarted","Data":"ca8dcda085588869bc68d5d6ae5b88009defa79d6b6dbf58f58aafe6f5563e6c"}
Jul 28 16:05:47.713427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.712998    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:05:47.714573 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.714541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8" event={"ID":"1f1a7ae2-8ee8-4891-95f3-71987273301e","Type":"ContainerStarted","Data":"1ae1f68bcf3e5d22349df67f31559c09c2dc4b9bf90fed7a0fd06c81668cc23e"}
Jul 28 16:05:47.714709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.714586    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:05:47.716029 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.716009    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr" event={"ID":"1f07b048-4cbb-4722-ba73-4cb6c353d1a2","Type":"ContainerStarted","Data":"6a4303cb8d9d8950e42f53127f8d0689e8b952c0563287a3f21a913d60475109"}
Jul 28 16:05:47.716151 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.716107    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:05:47.734126 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.734073    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-wc8z6" podStartSLOduration=2.697463095 podStartE2EDuration="6.734056329s" totalImagesPullingTime="4.036593234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:47.73223792 +0000 UTC m=+272.783041995" watchObservedRunningTime="2026-07-28 16:05:47.734056329 +0000 UTC m=+272.784860402"
Jul 28 16:05:47.749862 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.749800    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8" podStartSLOduration=2.054732507 podStartE2EDuration="5.749769673s" totalImagesPullingTime="3.695037166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:47.749647024 +0000 UTC m=+272.800451094" watchObservedRunningTime="2026-07-28 16:05:47.749769673 +0000 UTC m=+272.800573747"
Jul 28 16:05:47.771743 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:47.771688    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr" podStartSLOduration=1.4964654130000001 podStartE2EDuration="5.771666439s" totalImagesPullingTime="4.275201026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:05:47.769815966 +0000 UTC m=+272.820620051" watchObservedRunningTime="2026-07-28 16:05:47.771666439 +0000 UTC m=+272.822470514"
Jul 28 16:05:58.725512 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:05:58.725476    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gbjfr"
Jul 28 16:06:01.677938 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:01.677901    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-gdssp"
Jul 28 16:06:08.723164 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:08.723116    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-wc8z6"
Jul 28 16:06:08.723668 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:08.723218    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-nj8w8"
Jul 28 16:06:15.300307 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:15.300273    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:06:15.303113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:15.303086    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:06:15.306954 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:06:15.306921    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 16:07:00.115954 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.115910    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:00.118180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.118159    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.120662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.120610    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 16:07:00.120662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.120613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-p2h8r\""
Jul 28 16:07:00.121657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.121614    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 16:07:00.121657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.121645    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 16:07:00.131762 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.131728    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:00.155859 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.155826    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wtfmq"]
Jul 28 16:07:00.158248 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.158231    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.160841 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.160821    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lswf4\""
Jul 28 16:07:00.161007 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.160983    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 16:07:00.171253 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.171228    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wtfmq"]
Jul 28 16:07:00.179321 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.179293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ttsm\" (UniqueName: \"kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.179484 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.179346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78876\" (UniqueName: \"kubernetes.io/projected/2e991041-0b7c-40e2-9ad2-5006258e625d-kube-api-access-78876\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.179484 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.179390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2e991041-0b7c-40e2-9ad2-5006258e625d-data\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.179484 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.179460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.280743 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.280705    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2e991041-0b7c-40e2-9ad2-5006258e625d-data\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.280743 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.280755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.281017 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.280803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ttsm\" (UniqueName: \"kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.281017 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.280843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78876\" (UniqueName: \"kubernetes.io/projected/2e991041-0b7c-40e2-9ad2-5006258e625d-kube-api-access-78876\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.281149 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.281107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2e991041-0b7c-40e2-9ad2-5006258e625d-data\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.283458 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.283434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.291941 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.291913    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78876\" (UniqueName: \"kubernetes.io/projected/2e991041-0b7c-40e2-9ad2-5006258e625d-kube-api-access-78876\") pod \"seaweedfs-64568bcd49-wtfmq\" (UID: \"2e991041-0b7c-40e2-9ad2-5006258e625d\") " pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.292066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.291965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ttsm\" (UniqueName: \"kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm\") pod \"kserve-controller-manager-66f44fdc75-6sfrj\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") " pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.428665 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.428544    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:00.468235 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.468202    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:00.574750 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.574720    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:00.577286 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:07:00.577249    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494c7a27_2e68_4628_800d_24eb3a218ecf.slice/crio-48487a8101b9498adf89eb4e248905187557294e3601941199f5f978b3b4a4d7 WatchSource:0}: Error finding container 48487a8101b9498adf89eb4e248905187557294e3601941199f5f978b3b4a4d7: Status 404 returned error can't find the container with id 48487a8101b9498adf89eb4e248905187557294e3601941199f5f978b3b4a4d7
Jul 28 16:07:00.578686 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.578663    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:07:00.620649 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:00.620604    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wtfmq"]
Jul 28 16:07:00.622960 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:07:00.622926    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e991041_0b7c_40e2_9ad2_5006258e625d.slice/crio-4b5fcc15600e737086ac214ebd84e89f8532a5ca177c5c11f1994ed155b13db0 WatchSource:0}: Error finding container 4b5fcc15600e737086ac214ebd84e89f8532a5ca177c5c11f1994ed155b13db0: Status 404 returned error can't find the container with id 4b5fcc15600e737086ac214ebd84e89f8532a5ca177c5c11f1994ed155b13db0
Jul 28 16:07:01.034211 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:01.034173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wtfmq" event={"ID":"2e991041-0b7c-40e2-9ad2-5006258e625d","Type":"ContainerStarted","Data":"4b5fcc15600e737086ac214ebd84e89f8532a5ca177c5c11f1994ed155b13db0"}
Jul 28 16:07:01.035409 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:01.035382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" event={"ID":"494c7a27-2e68-4628-800d-24eb3a218ecf","Type":"ContainerStarted","Data":"48487a8101b9498adf89eb4e248905187557294e3601941199f5f978b3b4a4d7"}
Jul 28 16:07:05.059753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:05.059706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wtfmq" event={"ID":"2e991041-0b7c-40e2-9ad2-5006258e625d","Type":"ContainerStarted","Data":"cee0b75b029a0f2f39f81f73a6e6addcb9a1c5708e1113363ce989bbc02e1faa"}
Jul 28 16:07:05.060247 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:05.059864    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:05.076513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:05.076454    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wtfmq" podStartSLOduration=1.50355708 podStartE2EDuration="5.076430578s" totalImagesPullingTime="3.572873498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:05.075397526 +0000 UTC m=+350.126201600" watchObservedRunningTime="2026-07-28 16:07:05.076430578 +0000 UTC m=+350.127234653"
Jul 28 16:07:06.065523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:06.065481    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" event={"ID":"494c7a27-2e68-4628-800d-24eb3a218ecf","Type":"ContainerStarted","Data":"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"}
Jul 28 16:07:06.085233 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:06.085182    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" podStartSLOduration=0.790471241 podStartE2EDuration="6.085164734s" totalImagesPullingTime="5.294693493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:06.083358457 +0000 UTC m=+351.134162553" watchObservedRunningTime="2026-07-28 16:07:06.085164734 +0000 UTC m=+351.135968807"
Jul 28 16:07:07.070244 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:07.070206    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:11.068372 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:11.068337    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wtfmq"
Jul 28 16:07:38.079303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:38.079272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:44.898744 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:44.898698    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:44.899244 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:44.899047    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" podUID="494c7a27-2e68-4628-800d-24eb3a218ecf" containerName="manager" containerID="cri-o://38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77" gracePeriod=10
Jul 28 16:07:44.919731 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:44.919701    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-z98l2"]
Jul 28 16:07:44.922519 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:44.922502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:44.933537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:44.933504    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-z98l2"]
Jul 28 16:07:45.067240 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.067211    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-cert\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.067412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.067245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8wm\" (UniqueName: \"kubernetes.io/projected/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-kube-api-access-4s8wm\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.142159 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.142136    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:45.168323 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.168238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8wm\" (UniqueName: \"kubernetes.io/projected/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-kube-api-access-4s8wm\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.168485 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.168363    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-cert\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.170898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.170865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-cert\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.176476 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.176411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8wm\" (UniqueName: \"kubernetes.io/projected/73e741fd-312a-4ea3-b9f6-1dfaeae3e078-kube-api-access-4s8wm\") pod \"kserve-controller-manager-66f44fdc75-z98l2\" (UID: \"73e741fd-312a-4ea3-b9f6-1dfaeae3e078\") " pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.239969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.239932    2556 generic.go:356] "Generic (PLEG): container finished" podID="494c7a27-2e68-4628-800d-24eb3a218ecf" containerID="38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77" exitCode=0
Jul 28 16:07:45.240143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.239982    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj"
Jul 28 16:07:45.240143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.240022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" event={"ID":"494c7a27-2e68-4628-800d-24eb3a218ecf","Type":"ContainerDied","Data":"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"}
Jul 28 16:07:45.240143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.240080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-6sfrj" event={"ID":"494c7a27-2e68-4628-800d-24eb3a218ecf","Type":"ContainerDied","Data":"48487a8101b9498adf89eb4e248905187557294e3601941199f5f978b3b4a4d7"}
Jul 28 16:07:45.240143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.240104    2556 scope.go:117] "RemoveContainer" containerID="38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"
Jul 28 16:07:45.248487 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.248466    2556 scope.go:117] "RemoveContainer" containerID="38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"
Jul 28 16:07:45.248834 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:07:45.248813    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77\": container with ID starting with 38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77 not found: ID does not exist" containerID="38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"
Jul 28 16:07:45.248934 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.248841    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77"} err="failed to get container status \"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77\": rpc error: code = NotFound desc = could not find container \"38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77\": container with ID starting with 38e7c498af8daf8fcfeb550138c720fce29a16ef0e88ff0d57a9ff3f5b2dca77 not found: ID does not exist"
Jul 28 16:07:45.264164 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.264135    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:45.269210 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.269191    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert\") pod \"494c7a27-2e68-4628-800d-24eb3a218ecf\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") "
Jul 28 16:07:45.269271 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.269232    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ttsm\" (UniqueName: \"kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm\") pod \"494c7a27-2e68-4628-800d-24eb3a218ecf\" (UID: \"494c7a27-2e68-4628-800d-24eb3a218ecf\") "
Jul 28 16:07:45.271503 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.271478    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm" (OuterVolumeSpecName: "kube-api-access-9ttsm") pod "494c7a27-2e68-4628-800d-24eb3a218ecf" (UID: "494c7a27-2e68-4628-800d-24eb3a218ecf"). InnerVolumeSpecName "kube-api-access-9ttsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:07:45.271578 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.271543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert" (OuterVolumeSpecName: "cert") pod "494c7a27-2e68-4628-800d-24eb3a218ecf" (UID: "494c7a27-2e68-4628-800d-24eb3a218ecf"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:07:45.369992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.369964    2556 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/494c7a27-2e68-4628-800d-24eb3a218ecf-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:07:45.369992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.369992    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ttsm\" (UniqueName: \"kubernetes.io/projected/494c7a27-2e68-4628-800d-24eb3a218ecf-kube-api-access-9ttsm\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:07:45.399765 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.399709    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-z98l2"]
Jul 28 16:07:45.403548 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:07:45.403519    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e741fd_312a_4ea3_b9f6_1dfaeae3e078.slice/crio-c41cff651eed395f2e06bb24d1c5cdc23683d170c2ddc348e60f6d0e8d616493 WatchSource:0}: Error finding container c41cff651eed395f2e06bb24d1c5cdc23683d170c2ddc348e60f6d0e8d616493: Status 404 returned error can't find the container with id c41cff651eed395f2e06bb24d1c5cdc23683d170c2ddc348e60f6d0e8d616493
Jul 28 16:07:45.557809 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.557763    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:45.569041 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:45.569010    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66f44fdc75-6sfrj"]
Jul 28 16:07:46.246439 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:46.246405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2" event={"ID":"73e741fd-312a-4ea3-b9f6-1dfaeae3e078","Type":"ContainerStarted","Data":"2007e0f6376e08f1ea852c6072086428f4d38c562c4447886fb128b4ec7a21cd"}
Jul 28 16:07:46.246974 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:46.246467    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:07:46.246974 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:46.246490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2" event={"ID":"73e741fd-312a-4ea3-b9f6-1dfaeae3e078","Type":"ContainerStarted","Data":"c41cff651eed395f2e06bb24d1c5cdc23683d170c2ddc348e60f6d0e8d616493"}
Jul 28 16:07:46.265665 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:46.265591    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2" podStartSLOduration=1.8533426309999999 podStartE2EDuration="2.265573972s" totalImagesPullingTime="412.231341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:07:46.26414084 +0000 UTC m=+391.314944914" watchObservedRunningTime="2026-07-28 16:07:46.265573972 +0000 UTC m=+391.316378043"
Jul 28 16:07:47.462107 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:07:47.462071    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494c7a27-2e68-4628-800d-24eb3a218ecf" path="/var/lib/kubelet/pods/494c7a27-2e68-4628-800d-24eb3a218ecf/volumes"
Jul 28 16:08:03.140547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.140506    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fc455dfd-qmz4c"]
Jul 28 16:08:03.141095 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.141073    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494c7a27-2e68-4628-800d-24eb3a218ecf" containerName="manager"
Jul 28 16:08:03.141170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.141098    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="494c7a27-2e68-4628-800d-24eb3a218ecf" containerName="manager"
Jul 28 16:08:03.141223 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.141205    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="494c7a27-2e68-4628-800d-24eb3a218ecf" containerName="manager"
Jul 28 16:08:03.143561 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.143537    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.153219 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.153187    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fc455dfd-qmz4c"]
Jul 28 16:08:03.202521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202479    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202516    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-trusted-ca-bundle\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202540    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-oauth-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkl7\" (UniqueName: \"kubernetes.io/projected/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-kube-api-access-zlkl7\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-service-ca\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-oauth-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.202778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.202715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303061    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303094    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-trusted-ca-bundle\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303307 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-oauth-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303482 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkl7\" (UniqueName: \"kubernetes.io/projected/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-kube-api-access-zlkl7\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303587 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-service-ca\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303745 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303615    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-oauth-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.303745 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.303682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.304178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.304152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-trusted-ca-bundle\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.304436 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.304415    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-oauth-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.304502 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.304420    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.304538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.304512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-service-ca\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.305689 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.305668    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-serving-cert\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.305859 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.305843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-console-oauth-config\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.313463 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.313440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkl7\" (UniqueName: \"kubernetes.io/projected/1a928158-1ba4-461d-a0e4-e90ef7a2e4ec-kube-api-access-zlkl7\") pod \"console-8fc455dfd-qmz4c\" (UID: \"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec\") " pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.454479 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.454387    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:03.595545 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:03.595508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fc455dfd-qmz4c"]
Jul 28 16:08:03.598981 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:08:03.598955    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a928158_1ba4_461d_a0e4_e90ef7a2e4ec.slice/crio-7b2a24daf01cabd27c982f00a96597dcc010dd5188d18f5ed4b4f65aa590bbd2 WatchSource:0}: Error finding container 7b2a24daf01cabd27c982f00a96597dcc010dd5188d18f5ed4b4f65aa590bbd2: Status 404 returned error can't find the container with id 7b2a24daf01cabd27c982f00a96597dcc010dd5188d18f5ed4b4f65aa590bbd2
Jul 28 16:08:04.327258 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:04.327222    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc455dfd-qmz4c" event={"ID":"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec","Type":"ContainerStarted","Data":"c85122c3e5d1b8412b883e29c7d24a81884b64e425d1579f78ef141d04dfac3f"}
Jul 28 16:08:04.327685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:04.327276    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fc455dfd-qmz4c" event={"ID":"1a928158-1ba4-461d-a0e4-e90ef7a2e4ec","Type":"ContainerStarted","Data":"7b2a24daf01cabd27c982f00a96597dcc010dd5188d18f5ed4b4f65aa590bbd2"}
Jul 28 16:08:04.345566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:04.345508    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8fc455dfd-qmz4c" podStartSLOduration=1.34549045 podStartE2EDuration="1.34549045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:08:04.343161874 +0000 UTC m=+409.393965946" watchObservedRunningTime="2026-07-28 16:08:04.34549045 +0000 UTC m=+409.396294523"
Jul 28 16:08:13.454727 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:13.454690    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:13.455230 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:13.454745    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:13.461723 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:13.461697    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:14.376090 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:14.376062    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fc455dfd-qmz4c"
Jul 28 16:08:14.418382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:14.418342    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:08:17.256212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:17.256182    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66f44fdc75-z98l2"
Jul 28 16:08:18.479172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.479133    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-lcpnj"]
Jul 28 16:08:18.482065 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.482049    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.486012 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.485990    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-bpncc\""
Jul 28 16:08:18.487338 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.487318    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 16:08:18.497486 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.497458    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zjmc9"]
Jul 28 16:08:18.499824 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.499805    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:18.504347 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.504327    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6fq9g\""
Jul 28 16:08:18.504493 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.504454    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 16:08:18.522316 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.522265    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lcpnj"]
Jul 28 16:08:18.523890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.523865    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zjmc9"]
Jul 28 16:08:18.534717 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.534680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.534906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.534792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vrh\" (UniqueName: \"kubernetes.io/projected/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-kube-api-access-d4vrh\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.534906 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.534849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:18.535137 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.534912    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9kd8\" (UniqueName: \"kubernetes.io/projected/1c0a3517-fecd-480c-b8d6-772d91de5d35-kube-api-access-j9kd8\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:18.635728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.635692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9kd8\" (UniqueName: \"kubernetes.io/projected/1c0a3517-fecd-480c-b8d6-772d91de5d35-kube-api-access-j9kd8\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:18.635898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.635755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.635898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.635793    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vrh\" (UniqueName: \"kubernetes.io/projected/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-kube-api-access-d4vrh\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.635898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.635820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:18.635898 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:08:18.635888    2556 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 16:08:18.636038 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:08:18.635921    2556 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 16:08:18.636038 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:08:18.635961    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs podName:d9a93b2d-ea04-4781-8db0-eec6bb30f23c nodeName:}" failed. No retries permitted until 2026-07-28 16:08:19.135942962 +0000 UTC m=+424.186747013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs") pod "model-serving-api-7b954ffc95-lcpnj" (UID: "d9a93b2d-ea04-4781-8db0-eec6bb30f23c") : secret "model-serving-api-tls" not found
Jul 28 16:08:18.636038 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:08:18.635981    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert podName:1c0a3517-fecd-480c-b8d6-772d91de5d35 nodeName:}" failed. No retries permitted until 2026-07-28 16:08:19.135970915 +0000 UTC m=+424.186774972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert") pod "odh-model-controller-75d9cdbddf-zjmc9" (UID: "1c0a3517-fecd-480c-b8d6-772d91de5d35") : secret "odh-model-controller-webhook-cert" not found
Jul 28 16:08:18.652839 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.652806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vrh\" (UniqueName: \"kubernetes.io/projected/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-kube-api-access-d4vrh\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:18.658464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:18.658434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9kd8\" (UniqueName: \"kubernetes.io/projected/1c0a3517-fecd-480c-b8d6-772d91de5d35-kube-api-access-j9kd8\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:19.139084 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.139040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:19.139294 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.139132    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:19.142181 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.142152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9a93b2d-ea04-4781-8db0-eec6bb30f23c-tls-certs\") pod \"model-serving-api-7b954ffc95-lcpnj\" (UID: \"d9a93b2d-ea04-4781-8db0-eec6bb30f23c\") " pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:19.142319 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.142267    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c0a3517-fecd-480c-b8d6-772d91de5d35-cert\") pod \"odh-model-controller-75d9cdbddf-zjmc9\" (UID: \"1c0a3517-fecd-480c-b8d6-772d91de5d35\") " pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:19.392330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.392232    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:19.410050 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.410020    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:19.559781 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.559751    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-lcpnj"]
Jul 28 16:08:19.563151 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:08:19.563124    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a93b2d_ea04_4781_8db0_eec6bb30f23c.slice/crio-2076e97024a97960f699e6f8721ebe75a90fae0614c65851177ddf8c37ee5195 WatchSource:0}: Error finding container 2076e97024a97960f699e6f8721ebe75a90fae0614c65851177ddf8c37ee5195: Status 404 returned error can't find the container with id 2076e97024a97960f699e6f8721ebe75a90fae0614c65851177ddf8c37ee5195
Jul 28 16:08:19.575145 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:19.575120    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zjmc9"]
Jul 28 16:08:19.577130 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:08:19.577106    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a3517_fecd_480c_b8d6_772d91de5d35.slice/crio-55235a3b17642d3091e240d8a142c5bc6f4ebfeb1815fa1eb25e8368f7e84c03 WatchSource:0}: Error finding container 55235a3b17642d3091e240d8a142c5bc6f4ebfeb1815fa1eb25e8368f7e84c03: Status 404 returned error can't find the container with id 55235a3b17642d3091e240d8a142c5bc6f4ebfeb1815fa1eb25e8368f7e84c03
Jul 28 16:08:20.406183 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:20.406146    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lcpnj" event={"ID":"d9a93b2d-ea04-4781-8db0-eec6bb30f23c","Type":"ContainerStarted","Data":"2076e97024a97960f699e6f8721ebe75a90fae0614c65851177ddf8c37ee5195"}
Jul 28 16:08:20.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:20.408926    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9" event={"ID":"1c0a3517-fecd-480c-b8d6-772d91de5d35","Type":"ContainerStarted","Data":"55235a3b17642d3091e240d8a142c5bc6f4ebfeb1815fa1eb25e8368f7e84c03"}
Jul 28 16:08:23.433083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.432983    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9" event={"ID":"1c0a3517-fecd-480c-b8d6-772d91de5d35","Type":"ContainerStarted","Data":"ec90a900d26315a274f23b9baf8406322519d416536d04c7f7c90b1b3c50292a"}
Jul 28 16:08:23.433083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.433079    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:23.434634 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.434591    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-lcpnj" event={"ID":"d9a93b2d-ea04-4781-8db0-eec6bb30f23c","Type":"ContainerStarted","Data":"b177ce260909a3dfea722f06e51049dd243bbd372d1068b2f2f938623c12f2fc"}
Jul 28 16:08:23.434773 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.434684    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:23.473518 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.473462    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9" podStartSLOduration=1.865035569 podStartE2EDuration="5.473447069s" totalImagesPullingTime="3.6084115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:08:23.472846558 +0000 UTC m=+428.523650632" watchObservedRunningTime="2026-07-28 16:08:23.473447069 +0000 UTC m=+428.524251172"
Jul 28 16:08:23.520746 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:23.520678    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-lcpnj" podStartSLOduration=1.8889757390000002 podStartE2EDuration="5.520655901s" totalImagesPullingTime="3.631680162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:08:23.520452339 +0000 UTC m=+428.571256413" watchObservedRunningTime="2026-07-28 16:08:23.520655901 +0000 UTC m=+428.571459975"
Jul 28 16:08:34.441454 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:34.441421    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-zjmc9"
Jul 28 16:08:34.443491 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:34.443468    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-lcpnj"
Jul 28 16:08:39.438506 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.438401    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9d645d9c4-dmt4n" podUID="a99ef047-50af-4b22-bbdd-5d43277e6b53" containerName="console" containerID="cri-o://0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5" gracePeriod=15
Jul 28 16:08:39.699232 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.699165    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:08:39.833341 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833280    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833341 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833350    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833430    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833468    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ktsf\" (UniqueName: \"kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833496    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833529    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833615 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833582    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config\") pod \"a99ef047-50af-4b22-bbdd-5d43277e6b53\" (UID: \"a99ef047-50af-4b22-bbdd-5d43277e6b53\") "
Jul 28 16:08:39.833932 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833895    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:39.833987 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.833922    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:39.834052 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.834022    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca" (OuterVolumeSpecName: "service-ca") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:39.834326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.834303    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config" (OuterVolumeSpecName: "console-config") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:08:39.835875 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.835848    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:08:39.835875 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.835865    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:08:39.836023 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.835885    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf" (OuterVolumeSpecName: "kube-api-access-2ktsf") pod "a99ef047-50af-4b22-bbdd-5d43277e6b53" (UID: "a99ef047-50af-4b22-bbdd-5d43277e6b53"). InnerVolumeSpecName "kube-api-access-2ktsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:08:39.935250 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935207    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935250 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935242    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935250 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935251    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-trusted-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935250 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935260    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-oauth-serving-cert\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935268    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ktsf\" (UniqueName: \"kubernetes.io/projected/a99ef047-50af-4b22-bbdd-5d43277e6b53-kube-api-access-2ktsf\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935277    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a99ef047-50af-4b22-bbdd-5d43277e6b53-console-oauth-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:39.935516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:39.935286    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a99ef047-50af-4b22-bbdd-5d43277e6b53-service-ca\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:08:40.512496 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.512460    2556 generic.go:356] "Generic (PLEG): container finished" podID="a99ef047-50af-4b22-bbdd-5d43277e6b53" containerID="0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5" exitCode=0
Jul 28 16:08:40.512912 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.512510    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d645d9c4-dmt4n"
Jul 28 16:08:40.512912 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.512544    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d645d9c4-dmt4n" event={"ID":"a99ef047-50af-4b22-bbdd-5d43277e6b53","Type":"ContainerDied","Data":"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"}
Jul 28 16:08:40.512912 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.512600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d645d9c4-dmt4n" event={"ID":"a99ef047-50af-4b22-bbdd-5d43277e6b53","Type":"ContainerDied","Data":"cb274ca22f9e9a05c7c4ac554fd0a79713155e4f4d899800667b1e6323b639e2"}
Jul 28 16:08:40.512912 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.512639    2556 scope.go:117] "RemoveContainer" containerID="0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"
Jul 28 16:08:40.522200 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.522183    2556 scope.go:117] "RemoveContainer" containerID="0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"
Jul 28 16:08:40.522499 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:08:40.522479    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5\": container with ID starting with 0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5 not found: ID does not exist" containerID="0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"
Jul 28 16:08:40.522548 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.522511    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5"} err="failed to get container status \"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5\": rpc error: code = NotFound desc = could not find container \"0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5\": container with ID starting with 0eae2f1746a56aa4f0b28c1ddb3db75e85443801b826481aa4a05d8c23ce3da5 not found: ID does not exist"
Jul 28 16:08:40.538896 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.538858    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:08:40.544226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:40.544198    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d645d9c4-dmt4n"]
Jul 28 16:08:41.462335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:08:41.462301    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a99ef047-50af-4b22-bbdd-5d43277e6b53" path="/var/lib/kubelet/pods/a99ef047-50af-4b22-bbdd-5d43277e6b53/volumes"
Jul 28 16:10:06.867171 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.867126    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:10:06.867811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.867790    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99ef047-50af-4b22-bbdd-5d43277e6b53" containerName="console"
Jul 28 16:10:06.867811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.867811    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99ef047-50af-4b22-bbdd-5d43277e6b53" containerName="console"
Jul 28 16:10:06.867943 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.867924    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99ef047-50af-4b22-bbdd-5d43277e6b53" containerName="console"
Jul 28 16:10:06.870348 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.870326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:06.872966 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.872941    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 16:10:06.873929 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.873902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3692d-predictor-serving-cert\""
Jul 28 16:10:06.874052 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.873903    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-k6dfd\""
Jul 28 16:10:06.874052 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.873906    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 16:10:06.874052 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.873906    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3692d-kube-rbac-proxy-sar-config\""
Jul 28 16:10:06.882849 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.882825    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:10:06.972228 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.972191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs6gm\" (UniqueName: \"kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:06.972427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.972248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:06.972427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:06.972313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.073955 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.073908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs6gm\" (UniqueName: \"kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.074131 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.073986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.074131 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.074026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.074923 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.074841    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.077091 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.077066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.084473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.084446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs6gm\" (UniqueName: \"kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm\") pod \"success-200-isvc-3692d-predictor-5b97fbf46c-6tfht\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") " pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.182662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.182568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:07.338842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.338807    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:10:07.342598 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:10:07.342565    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7d8d41_d9ba_4512_9540_fd814b045f09.slice/crio-2bc70be3b5c0b9b2d38ce01c45ee7b7120ee966d4e58e89d9a0e133463b11e47 WatchSource:0}: Error finding container 2bc70be3b5c0b9b2d38ce01c45ee7b7120ee966d4e58e89d9a0e133463b11e47: Status 404 returned error can't find the container with id 2bc70be3b5c0b9b2d38ce01c45ee7b7120ee966d4e58e89d9a0e133463b11e47
Jul 28 16:10:07.388035 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.388003    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:10:07.391331 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.391313    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.393868 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.393848    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3692d-kube-rbac-proxy-sar-config\""
Jul 28 16:10:07.393988 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.393846    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3692d-predictor-serving-cert\""
Jul 28 16:10:07.402073 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.402050    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:10:07.477116 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.477030    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.477116 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.477079    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzm5\" (UniqueName: \"kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.477297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.477190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.578108 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.578068    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.578281 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.578195    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.578281 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.578231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzm5\" (UniqueName: \"kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.578933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.578905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.581072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.581048    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.588437 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.588413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzm5\" (UniqueName: \"kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5\") pod \"error-404-isvc-3692d-predictor-97c56c7cd-jjssk\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") " pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.703269 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.703230    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:07.845318 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.845195    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:10:07.849446 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:10:07.849422    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d09874_be09_45a1_8c27_a26033be48a6.slice/crio-e04ddcf914a5291f5ee6316bda72786c33b4767ce24e6884121adb743a271941 WatchSource:0}: Error finding container e04ddcf914a5291f5ee6316bda72786c33b4767ce24e6884121adb743a271941: Status 404 returned error can't find the container with id e04ddcf914a5291f5ee6316bda72786c33b4767ce24e6884121adb743a271941
Jul 28 16:10:07.912007 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.911944    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerStarted","Data":"e04ddcf914a5291f5ee6316bda72786c33b4767ce24e6884121adb743a271941"}
Jul 28 16:10:07.913304 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:07.913263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerStarted","Data":"2bc70be3b5c0b9b2d38ce01c45ee7b7120ee966d4e58e89d9a0e133463b11e47"}
Jul 28 16:10:24.018963 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:24.018864    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerStarted","Data":"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"}
Jul 28 16:10:24.020385 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:24.020357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerStarted","Data":"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"}
Jul 28 16:10:27.037704 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.037657    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerStarted","Data":"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"}
Jul 28 16:10:27.038160 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.037741    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:27.038160 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.037900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:27.039122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.039086    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:10:27.039556 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.039537    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerStarted","Data":"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"}
Jul 28 16:10:27.039744 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.039732    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:27.060058 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.059994    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podStartSLOduration=1.494794393 podStartE2EDuration="20.059964556s" totalImagesPullingTime="18.565170163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:10:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:10:27.057705149 +0000 UTC m=+552.108509216" watchObservedRunningTime="2026-07-28 16:10:27.059964556 +0000 UTC m=+552.110768633"
Jul 28 16:10:27.077870 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:27.077811    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podStartSLOduration=1.993861684 podStartE2EDuration="21.077797434s" totalImagesPullingTime="19.08393575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:10:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:10:27.076374913 +0000 UTC m=+552.127178990" watchObservedRunningTime="2026-07-28 16:10:27.077797434 +0000 UTC m=+552.128601506"
Jul 28 16:10:28.045863 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:28.045808    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:10:28.046422 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:28.046038    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:28.047398 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:28.047369    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:10:29.050657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:29.050588    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:10:33.051823 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:33.051795    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:10:33.052312 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:33.052287    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:10:34.057873 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:34.057845    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:10:34.058327 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:34.058300    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:10:43.052407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:43.052352    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:10:44.058564 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:44.058516    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:10:53.052357 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:53.052317    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:10:54.058291 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:10:54.058253    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:11:03.053214 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:03.053168    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:11:04.058613 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:04.058574    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:11:13.053287 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:13.053248    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:11:14.059485 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:14.059455    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:11:15.330114 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:15.330082    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:11:15.331235 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:15.331208    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:11:26.897155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.897116    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:11:26.900854 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.900836    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:26.905059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.905028    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3692d-kube-rbac-proxy-sar-config\""
Jul 28 16:11:26.905195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.905028    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-3692d-serving-cert\""
Jul 28 16:11:26.905514 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.905496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:26.905557 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.905533    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:26.912345 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:26.912317    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:11:27.006249 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.006206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.006411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.006294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.006411 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:27.006329    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-3692d-serving-cert: secret "switch-graph-3692d-serving-cert" not found
Jul 28 16:11:27.006483 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:27.006412    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls podName:6a379b69-6194-4300-be5f-dc427636e814 nodeName:}" failed. No retries permitted until 2026-07-28 16:11:27.506395769 +0000 UTC m=+612.557199823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls") pod "switch-graph-3692d-d8b5895b7-wbvlr" (UID: "6a379b69-6194-4300-be5f-dc427636e814") : secret "switch-graph-3692d-serving-cert" not found
Jul 28 16:11:27.006876 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.006859    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.510461 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.510431    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.513055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.513032    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") pod \"switch-graph-3692d-d8b5895b7-wbvlr\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") " pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.812213 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.812097    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:27.952036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:27.951988    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:11:27.955090 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:11:27.955056    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a379b69_6194_4300_be5f_dc427636e814.slice/crio-dc2e24fe6e5e20981906d3ba51fdd2f48b748697cd8f5c33106cfff98a473515 WatchSource:0}: Error finding container dc2e24fe6e5e20981906d3ba51fdd2f48b748697cd8f5c33106cfff98a473515: Status 404 returned error can't find the container with id dc2e24fe6e5e20981906d3ba51fdd2f48b748697cd8f5c33106cfff98a473515
Jul 28 16:11:28.332819 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:28.332783    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" event={"ID":"6a379b69-6194-4300-be5f-dc427636e814","Type":"ContainerStarted","Data":"dc2e24fe6e5e20981906d3ba51fdd2f48b748697cd8f5c33106cfff98a473515"}
Jul 28 16:11:31.353040 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:31.353003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" event={"ID":"6a379b69-6194-4300-be5f-dc427636e814","Type":"ContainerStarted","Data":"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"}
Jul 28 16:11:31.353442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:31.353112    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:31.369759 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:31.369704    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podStartSLOduration=2.574840209 podStartE2EDuration="5.369688401s" totalImagesPullingTime="2.794848192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:11:31.369247902 +0000 UTC m=+616.420051976" watchObservedRunningTime="2026-07-28 16:11:31.369688401 +0000 UTC m=+616.420492476"
Jul 28 16:11:37.363259 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:37.363187    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:41.021286 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.021253    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:11:41.021717 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.021463    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" containerID="cri-o://3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730" gracePeriod=30
Jul 28 16:11:41.194381 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.194340    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:11:41.194701 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.194673    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" containerID="cri-o://764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875" gracePeriod=30
Jul 28 16:11:41.194791 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.194696    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kube-rbac-proxy" containerID="cri-o://d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4" gracePeriod=30
Jul 28 16:11:41.242988 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.242954    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:11:41.245597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.245574    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.248312 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.248288    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-021cc-predictor-serving-cert\""
Jul 28 16:11:41.248476 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.248296    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-021cc-kube-rbac-proxy-sar-config\""
Jul 28 16:11:41.257499 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.257473    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:11:41.323419 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.323382    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:11:41.326896 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.326870    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.329674 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.329648    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-021cc-predictor-serving-cert\""
Jul 28 16:11:41.330023 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.330000    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-021cc-kube-rbac-proxy-sar-config\""
Jul 28 16:11:41.331637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.331604    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.331723 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.331668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.331814 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.331800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57hb\" (UniqueName: \"kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.338246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.338219    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:11:41.401186 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.401150    2556 generic.go:356] "Generic (PLEG): container finished" podID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerID="d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4" exitCode=2
Jul 28 16:11:41.401365 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.401215    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerDied","Data":"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"}
Jul 28 16:11:41.432780 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432750    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57hb\" (UniqueName: \"kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.432971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.432971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.432971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.432971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432939    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.433268 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.432994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vhb\" (UniqueName: \"kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.433658 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.433615    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.436268 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.436206    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.447680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.447646    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57hb\" (UniqueName: \"kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb\") pod \"success-200-isvc-021cc-predictor-7b688f7777-fsnkl\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") " pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.474723 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.474686    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:11:41.474920 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.474895    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" containerID="cri-o://f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa" gracePeriod=30
Jul 28 16:11:41.475014 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.474935    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kube-rbac-proxy" containerID="cri-o://0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03" gracePeriod=30
Jul 28 16:11:41.534174 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.534137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.534379 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.534224    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vhb\" (UniqueName: \"kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.534379 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:41.534328    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-021cc-predictor-serving-cert: secret "error-404-isvc-021cc-predictor-serving-cert" not found
Jul 28 16:11:41.534497 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:41.534421    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls podName:4f39f911-e715-46b1-b23e-50899a865a99 nodeName:}" failed. No retries permitted until 2026-07-28 16:11:42.034397322 +0000 UTC m=+627.085201389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls") pod "error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" (UID: "4f39f911-e715-46b1-b23e-50899a865a99") : secret "error-404-isvc-021cc-predictor-serving-cert" not found
Jul 28 16:11:41.534497 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.534341    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.535045 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.535025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.545540 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.545509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vhb\" (UniqueName: \"kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:41.557372 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.557340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:41.702177 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:41.702133    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:11:41.704591 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:11:41.704562    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5deadc31_746f_49fe_a654_d1f27457af11.slice/crio-816437aa96c887d7606a44c0cac10b9da252f7f1235c24a9234494075ec7cec4 WatchSource:0}: Error finding container 816437aa96c887d7606a44c0cac10b9da252f7f1235c24a9234494075ec7cec4: Status 404 returned error can't find the container with id 816437aa96c887d7606a44c0cac10b9da252f7f1235c24a9234494075ec7cec4
Jul 28 16:11:42.038382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.038342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:42.040964 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.040938    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") pod \"error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") " pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:42.238674 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.238600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:42.362353 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.362310    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:11:42.384022 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.383920    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:11:42.386643 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:11:42.386587    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f39f911_e715_46b1_b23e_50899a865a99.slice/crio-629fc131974c52defdd2dab0e4a53cbc850840c4e249d4c8774a1ac781c6da0b WatchSource:0}: Error finding container 629fc131974c52defdd2dab0e4a53cbc850840c4e249d4c8774a1ac781c6da0b: Status 404 returned error can't find the container with id 629fc131974c52defdd2dab0e4a53cbc850840c4e249d4c8774a1ac781c6da0b
Jul 28 16:11:42.409423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.409379    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerStarted","Data":"629fc131974c52defdd2dab0e4a53cbc850840c4e249d4c8774a1ac781c6da0b"}
Jul 28 16:11:42.411330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.411301    2556 generic.go:356] "Generic (PLEG): container finished" podID="48d09874-be09-45a1-8c27-a26033be48a6" containerID="0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03" exitCode=2
Jul 28 16:11:42.411442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.411349    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerDied","Data":"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"}
Jul 28 16:11:42.413263 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.413241    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerStarted","Data":"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"}
Jul 28 16:11:42.413377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.413286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerStarted","Data":"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"}
Jul 28 16:11:42.413377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.413306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerStarted","Data":"816437aa96c887d7606a44c0cac10b9da252f7f1235c24a9234494075ec7cec4"}
Jul 28 16:11:42.413473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.413448    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:42.413527 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.413508    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:42.414711 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.414688    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:11:42.436049 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:42.435992    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podStartSLOduration=1.435975504 podStartE2EDuration="1.435975504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:11:42.435111403 +0000 UTC m=+627.485915477" watchObservedRunningTime="2026-07-28 16:11:42.435975504 +0000 UTC m=+627.486779576"
Jul 28 16:11:43.046219 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.046168    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 16:11:43.052396 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.052361    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 16:11:43.420170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.420132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerStarted","Data":"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"}
Jul 28 16:11:43.420423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.420201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerStarted","Data":"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"}
Jul 28 16:11:43.420512 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.420474    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:11:43.420556 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.420511    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:43.420593 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.420561    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:43.421720 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.421693    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:11:43.441359 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:43.441297    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podStartSLOduration=2.441273553 podStartE2EDuration="2.441273553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:11:43.439576083 +0000 UTC m=+628.490380158" watchObservedRunningTime="2026-07-28 16:11:43.441273553 +0000 UTC m=+628.492077631"
Jul 28 16:11:44.051639 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:44.051584    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 16:11:44.058609 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:44.058576    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 16:11:44.425336 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:44.425296    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:11:45.261396 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.261368    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:11:45.334731 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.334706    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:11:45.369984 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.369939    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config\") pod \"bc7d8d41-d9ba-4512-9540-fd814b045f09\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") "
Jul 28 16:11:45.370170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.370016    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls\") pod \"bc7d8d41-d9ba-4512-9540-fd814b045f09\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") "
Jul 28 16:11:45.370170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.370105    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs6gm\" (UniqueName: \"kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm\") pod \"bc7d8d41-d9ba-4512-9540-fd814b045f09\" (UID: \"bc7d8d41-d9ba-4512-9540-fd814b045f09\") "
Jul 28 16:11:45.370344 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.370313    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bc7d8d41-d9ba-4512-9540-fd814b045f09" (UID: "bc7d8d41-d9ba-4512-9540-fd814b045f09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:11:45.372403 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.372379    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc7d8d41-d9ba-4512-9540-fd814b045f09" (UID: "bc7d8d41-d9ba-4512-9540-fd814b045f09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:11:45.372473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.372438    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm" (OuterVolumeSpecName: "kube-api-access-gs6gm") pod "bc7d8d41-d9ba-4512-9540-fd814b045f09" (UID: "bc7d8d41-d9ba-4512-9540-fd814b045f09"). InnerVolumeSpecName "kube-api-access-gs6gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:11:45.431636 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.431588    2556 generic.go:356] "Generic (PLEG): container finished" podID="48d09874-be09-45a1-8c27-a26033be48a6" containerID="f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa" exitCode=0
Jul 28 16:11:45.431830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.431659    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"
Jul 28 16:11:45.431830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.431675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerDied","Data":"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"}
Jul 28 16:11:45.431830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.431749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk" event={"ID":"48d09874-be09-45a1-8c27-a26033be48a6","Type":"ContainerDied","Data":"e04ddcf914a5291f5ee6316bda72786c33b4767ce24e6884121adb743a271941"}
Jul 28 16:11:45.431830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.431786    2556 scope.go:117] "RemoveContainer" containerID="0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"
Jul 28 16:11:45.433363 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.433343    2556 generic.go:356] "Generic (PLEG): container finished" podID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerID="764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875" exitCode=0
Jul 28 16:11:45.433464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.433381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"
Jul 28 16:11:45.433464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.433395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerDied","Data":"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"}
Jul 28 16:11:45.433464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.433433    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht" event={"ID":"bc7d8d41-d9ba-4512-9540-fd814b045f09","Type":"ContainerDied","Data":"2bc70be3b5c0b9b2d38ce01c45ee7b7120ee966d4e58e89d9a0e133463b11e47"}
Jul 28 16:11:45.440882 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.440851    2556 scope.go:117] "RemoveContainer" containerID="f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"
Jul 28 16:11:45.451175 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.451150    2556 scope.go:117] "RemoveContainer" containerID="0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"
Jul 28 16:11:45.451501 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:45.451484    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03\": container with ID starting with 0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03 not found: ID does not exist" containerID="0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"
Jul 28 16:11:45.451566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.451511    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03"} err="failed to get container status \"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03\": rpc error: code = NotFound desc = could not find container \"0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03\": container with ID starting with 0ca832adf44d77889b68d67a3c0f4f38347dbc9b0844aabffb6504cbd2df5b03 not found: ID does not exist"
Jul 28 16:11:45.451566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.451529    2556 scope.go:117] "RemoveContainer" containerID="f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"
Jul 28 16:11:45.451826 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:45.451806    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa\": container with ID starting with f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa not found: ID does not exist" containerID="f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"
Jul 28 16:11:45.451889 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.451836    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa"} err="failed to get container status \"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa\": rpc error: code = NotFound desc = could not find container \"f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa\": container with ID starting with f1e4c333607d91a73de3a2ecebafeb8f6011f67fedb59f2190b45f9fb3755efa not found: ID does not exist"
Jul 28 16:11:45.451889 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.451862    2556 scope.go:117] "RemoveContainer" containerID="d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"
Jul 28 16:11:45.460541 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.460509    2556 scope.go:117] "RemoveContainer" containerID="764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"
Jul 28 16:11:45.467457 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.467427    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:11:45.469266 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.469242    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3692d-predictor-5b97fbf46c-6tfht"]
Jul 28 16:11:45.469978 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.469960    2556 scope.go:117] "RemoveContainer" containerID="d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"
Jul 28 16:11:45.470280 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:45.470261    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4\": container with ID starting with d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4 not found: ID does not exist" containerID="d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"
Jul 28 16:11:45.470328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470288    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4"} err="failed to get container status \"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4\": rpc error: code = NotFound desc = could not find container \"d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4\": container with ID starting with d3876e52cac5997167d3c3e192788f8520fffd0ff5562b62147fb49c8e1d14a4 not found: ID does not exist"
Jul 28 16:11:45.470328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470307    2556 scope.go:117] "RemoveContainer" containerID="764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"
Jul 28 16:11:45.470529 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:11:45.470512    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875\": container with ID starting with 764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875 not found: ID does not exist" containerID="764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"
Jul 28 16:11:45.470585 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470533    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875"} err="failed to get container status \"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875\": rpc error: code = NotFound desc = could not find container \"764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875\": container with ID starting with 764f3fd78d898a25ea2e80172a8ca49e8cba97129e5870de1986d843b93bd875 not found: ID does not exist"
Jul 28 16:11:45.470585 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470569    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgzm5\" (UniqueName: \"kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5\") pod \"48d09874-be09-45a1-8c27-a26033be48a6\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") "
Jul 28 16:11:45.470707 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config\") pod \"48d09874-be09-45a1-8c27-a26033be48a6\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") "
Jul 28 16:11:45.470764 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470719    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls\") pod \"48d09874-be09-45a1-8c27-a26033be48a6\" (UID: \"48d09874-be09-45a1-8c27-a26033be48a6\") "
Jul 28 16:11:45.470981 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470967    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs6gm\" (UniqueName: \"kubernetes.io/projected/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-api-access-gs6gm\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.471057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.470987    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bc7d8d41-d9ba-4512-9540-fd814b045f09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.471057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.471001    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc7d8d41-d9ba-4512-9540-fd814b045f09-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.471057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.471020    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48d09874-be09-45a1-8c27-a26033be48a6" (UID: "48d09874-be09-45a1-8c27-a26033be48a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:11:45.472778 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.472759    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48d09874-be09-45a1-8c27-a26033be48a6" (UID: "48d09874-be09-45a1-8c27-a26033be48a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:11:45.472848 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.472829    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5" (OuterVolumeSpecName: "kube-api-access-tgzm5") pod "48d09874-be09-45a1-8c27-a26033be48a6" (UID: "48d09874-be09-45a1-8c27-a26033be48a6"). InnerVolumeSpecName "kube-api-access-tgzm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:11:45.571664 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.571535    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48d09874-be09-45a1-8c27-a26033be48a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.571664 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.571569    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48d09874-be09-45a1-8c27-a26033be48a6-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.571664 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.571586    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzm5\" (UniqueName: \"kubernetes.io/projected/48d09874-be09-45a1-8c27-a26033be48a6-kube-api-access-tgzm5\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:11:45.755540 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.755499    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:11:45.759401 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:45.759375    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3692d-predictor-97c56c7cd-jjssk"]
Jul 28 16:11:47.362273 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:47.362233    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:11:47.462448 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:47.462412    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d09874-be09-45a1-8c27-a26033be48a6" path="/var/lib/kubelet/pods/48d09874-be09-45a1-8c27-a26033be48a6/volumes"
Jul 28 16:11:47.462889 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:47.462875    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" path="/var/lib/kubelet/pods/bc7d8d41-d9ba-4512-9540-fd814b045f09/volumes"
Jul 28 16:11:48.425223 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:48.425194    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:11:48.425859 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:48.425830    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:11:49.430903 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:49.430872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:11:49.431372 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:49.431346    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:11:52.361516 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:52.361480    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:11:52.361939 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:52.361595    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:11:57.362155 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:57.362115    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:11:58.426480 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:58.426436    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:11:59.431542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:11:59.431503    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:12:02.361612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:02.361569    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:07.361481 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:07.361435    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:08.426711 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:08.426669    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:12:09.431933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:09.431891    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:12:11.217806 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.217777    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:12:11.310789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.310698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle\") pod \"6a379b69-6194-4300-be5f-dc427636e814\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") "
Jul 28 16:12:11.310789 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.310785    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") pod \"6a379b69-6194-4300-be5f-dc427636e814\" (UID: \"6a379b69-6194-4300-be5f-dc427636e814\") "
Jul 28 16:12:11.311023 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.310999    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6a379b69-6194-4300-be5f-dc427636e814" (UID: "6a379b69-6194-4300-be5f-dc427636e814"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:12:11.312947 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.312923    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a379b69-6194-4300-be5f-dc427636e814" (UID: "6a379b69-6194-4300-be5f-dc427636e814"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:12:11.412038 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.411988    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a379b69-6194-4300-be5f-dc427636e814-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:12:11.412038 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.412034    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a379b69-6194-4300-be5f-dc427636e814-openshift-service-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:12:11.572647 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.572534    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a379b69-6194-4300-be5f-dc427636e814" containerID="3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730" exitCode=137
Jul 28 16:12:11.572647 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.572580    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"
Jul 28 16:12:11.572647 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.572615    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" event={"ID":"6a379b69-6194-4300-be5f-dc427636e814","Type":"ContainerDied","Data":"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"}
Jul 28 16:12:11.572908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.572680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr" event={"ID":"6a379b69-6194-4300-be5f-dc427636e814","Type":"ContainerDied","Data":"dc2e24fe6e5e20981906d3ba51fdd2f48b748697cd8f5c33106cfff98a473515"}
Jul 28 16:12:11.572908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.572703    2556 scope.go:117] "RemoveContainer" containerID="3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"
Jul 28 16:12:11.583948 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.583535    2556 scope.go:117] "RemoveContainer" containerID="3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"
Jul 28 16:12:11.584066 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:12:11.583997    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730\": container with ID starting with 3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730 not found: ID does not exist" containerID="3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"
Jul 28 16:12:11.584066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.584032    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730"} err="failed to get container status \"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730\": rpc error: code = NotFound desc = could not find container \"3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730\": container with ID starting with 3b9b21da7c7258ca4fcdbcdd2f275e27bc4d87e88d987e3c59a7ede3dbb46730 not found: ID does not exist"
Jul 28 16:12:11.598596 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.598562    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:12:11.606375 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:11.606352    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-3692d-d8b5895b7-wbvlr"]
Jul 28 16:12:13.463730 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:13.463699    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a379b69-6194-4300-be5f-dc427636e814" path="/var/lib/kubelet/pods/6a379b69-6194-4300-be5f-dc427636e814/volumes"
Jul 28 16:12:18.426001 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:18.425960    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:12:19.431941 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:19.431901    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:12:28.426993 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:28.426958    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:12:29.433301 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:29.433270    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:12:36.949303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949266    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:12:36.949910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949869    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d"
Jul 28 16:12:36.949910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949890    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d"
Jul 28 16:12:36.949910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949912    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949921    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949939    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949947    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949956    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949961    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949968    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.949975    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.950071    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.950081    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7d8d41-d9ba-4512-9540-fd814b045f09" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.950087    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kube-rbac-proxy"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.950093    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d09874-be09-45a1-8c27-a26033be48a6" containerName="kserve-container"
Jul 28 16:12:36.950113 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.950101    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a379b69-6194-4300-be5f-dc427636e814" containerName="switch-graph-3692d"
Jul 28 16:12:36.954903 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.954882    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:36.959032 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.959003    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 16:12:36.959290 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.959275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 16:12:36.966297 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:36.966270    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:12:37.026721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.026680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.026934 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.026795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.127385 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.127345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.127553 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.127415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.127553 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:12:37.127514    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jul 28 16:12:37.127656 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:12:37.127580    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls podName:58640b38-9c01-4db7-aaee-5a5b1b665374 nodeName:}" failed. No retries permitted until 2026-07-28 16:12:37.627564299 +0000 UTC m=+682.678368353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls") pod "model-chainer-fff6dfdd4-k75lz" (UID: "58640b38-9c01-4db7-aaee-5a5b1b665374") : secret "model-chainer-serving-cert" not found
Jul 28 16:12:37.128057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.128034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.631900 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.631854    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.634580 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.634543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") pod \"model-chainer-fff6dfdd4-k75lz\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") " pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:37.867013 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:37.866977    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:38.018214 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:38.018174    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:12:38.022248 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:12:38.022222    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58640b38_9c01_4db7_aaee_5a5b1b665374.slice/crio-e3fc505fbeab9e995a4a13326198cf87ebd27eda38dfbce9d68f39d98019471b WatchSource:0}: Error finding container e3fc505fbeab9e995a4a13326198cf87ebd27eda38dfbce9d68f39d98019471b: Status 404 returned error can't find the container with id e3fc505fbeab9e995a4a13326198cf87ebd27eda38dfbce9d68f39d98019471b
Jul 28 16:12:38.701398 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:38.701364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" event={"ID":"58640b38-9c01-4db7-aaee-5a5b1b665374","Type":"ContainerStarted","Data":"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"}
Jul 28 16:12:38.701576 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:38.701412    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" event={"ID":"58640b38-9c01-4db7-aaee-5a5b1b665374","Type":"ContainerStarted","Data":"e3fc505fbeab9e995a4a13326198cf87ebd27eda38dfbce9d68f39d98019471b"}
Jul 28 16:12:38.701576 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:38.701463    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:38.729862 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:38.729813    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podStartSLOduration=2.729795942 podStartE2EDuration="2.729795942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:12:38.728042166 +0000 UTC m=+683.778846284" watchObservedRunningTime="2026-07-28 16:12:38.729795942 +0000 UTC m=+683.780600085"
Jul 28 16:12:41.310712 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.310675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:12:41.314435 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.314409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.318354 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.318326    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-021cc-kube-rbac-proxy-sar-config\""
Jul 28 16:12:41.318487 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.318326    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-021cc-serving-cert\""
Jul 28 16:12:41.326958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.326926    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:12:41.366121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.366080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.366319 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.366166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.467190 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.467154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.467359 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.467219    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.467934 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.467868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.469746 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.469722    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls\") pod \"switch-graph-021cc-6486fdbd5-9fr9n\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") " pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.625061 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.625019    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:41.768051 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:41.767871    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:12:41.770369 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:12:41.770341    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a1d87c_0187_4d95_b98b_9ecd12661654.slice/crio-d15a9c1102e1ef1df08aabd61efc3317f343beffa36e29a750a84dcb4a167bca WatchSource:0}: Error finding container d15a9c1102e1ef1df08aabd61efc3317f343beffa36e29a750a84dcb4a167bca: Status 404 returned error can't find the container with id d15a9c1102e1ef1df08aabd61efc3317f343beffa36e29a750a84dcb4a167bca
Jul 28 16:12:42.722964 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:42.722930    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" event={"ID":"15a1d87c-0187-4d95-b98b-9ecd12661654","Type":"ContainerStarted","Data":"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"}
Jul 28 16:12:42.723469 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:42.722981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" event={"ID":"15a1d87c-0187-4d95-b98b-9ecd12661654","Type":"ContainerStarted","Data":"d15a9c1102e1ef1df08aabd61efc3317f343beffa36e29a750a84dcb4a167bca"}
Jul 28 16:12:42.723469 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:42.723038    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:42.745751 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:42.745650    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podStartSLOduration=1.745600328 podStartE2EDuration="1.745600328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:12:42.743801228 +0000 UTC m=+687.794605301" watchObservedRunningTime="2026-07-28 16:12:42.745600328 +0000 UTC m=+687.796404403"
Jul 28 16:12:44.713051 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:44.713021    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:12:48.733965 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:48.733937    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:12:51.052837 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.052797    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:12:51.053205 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.053093    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" containerID="cri-o://f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d" gracePeriod=30
Jul 28 16:12:51.258042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.258010    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:12:51.261955 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.261932    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.264658 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.264612    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0402e-kube-rbac-proxy-sar-config\""
Jul 28 16:12:51.264771 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.264613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0402e-predictor-serving-cert\""
Jul 28 16:12:51.273846 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.273816    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:12:51.353160 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.353120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.353348 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.353189    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.353348 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.353222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.415072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.415037    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:12:51.418812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.418792    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.422796 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.422776    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0402e-kube-rbac-proxy-sar-config\""
Jul 28 16:12:51.422902 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.422842    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0402e-predictor-serving-cert\""
Jul 28 16:12:51.435544 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.435515    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:12:51.453891 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.453850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.454092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.453916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.454092 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.453943    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.455142 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.455107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.457076 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.457015    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.464321 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.464289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n\") pod \"success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") " pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.554759 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.554720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.554759 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.554761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.555006 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.554793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtds\" (UniqueName: \"kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.574143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.574108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:51.656094 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.656061    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.656833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.656101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.656833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.656131    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtds\" (UniqueName: \"kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.656967 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.656942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.658972 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.658892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.670333 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.670295    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtds\" (UniqueName: \"kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds\") pod \"error-404-isvc-0402e-predictor-74b6c76776-g66wb\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") " pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.729013 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.728979    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:51.734309 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.734283    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:12:51.736391 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:12:51.736360    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1de457f_a67f_40b2_98de_c159c23a9a0a.slice/crio-4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc WatchSource:0}: Error finding container 4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc: Status 404 returned error can't find the container with id 4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc
Jul 28 16:12:51.769010 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.768979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerStarted","Data":"4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc"}
Jul 28 16:12:51.885680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:51.885574    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:12:51.890069 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:12:51.890029    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675be705_9def_42cb_983c_5a12e49fe04a.slice/crio-36d8b14b86d447c4f2a84c8e5d2429991215694f8696e6a050282ebe6173db17 WatchSource:0}: Error finding container 36d8b14b86d447c4f2a84c8e5d2429991215694f8696e6a050282ebe6173db17: Status 404 returned error can't find the container with id 36d8b14b86d447c4f2a84c8e5d2429991215694f8696e6a050282ebe6173db17
Jul 28 16:12:52.775517 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.775475    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerStarted","Data":"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"}
Jul 28 16:12:52.776043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.775573    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:52.776043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.775636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerStarted","Data":"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"}
Jul 28 16:12:52.776043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.775671    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:52.776043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.775720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerStarted","Data":"36d8b14b86d447c4f2a84c8e5d2429991215694f8696e6a050282ebe6173db17"}
Jul 28 16:12:52.777066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.777039    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:12:52.777783 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.777754    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerStarted","Data":"f6ae9ba6dba11fb73c7bc3001d58f2f0975cf9e9792a8a388a3115e1d06b37ae"}
Jul 28 16:12:52.777885 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.777817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerStarted","Data":"3065835da5e0295e1dc21bf791387b846bf3de3852af301db2bda4838733fa1e"}
Jul 28 16:12:52.777885 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.777852    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:52.777885 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.777882    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:52.779106 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.779085    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:12:52.801599 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.801541    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podStartSLOduration=1.8015239379999999 podStartE2EDuration="1.801523938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:12:52.798882498 +0000 UTC m=+697.849686571" watchObservedRunningTime="2026-07-28 16:12:52.801523938 +0000 UTC m=+697.852328015"
Jul 28 16:12:52.837365 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:52.837307    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podStartSLOduration=1.837289892 podStartE2EDuration="1.837289892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:12:52.834870409 +0000 UTC m=+697.885674482" watchObservedRunningTime="2026-07-28 16:12:52.837289892 +0000 UTC m=+697.888093965"
Jul 28 16:12:53.782993 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:53.782941    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:12:53.783421 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:53.782941    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:12:54.711166 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:54.711122    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:12:58.787703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:58.787671    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:12:58.788157 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:58.787966    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:12:58.788157 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:58.788090    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:12:58.788428 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:58.788401    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:12:59.710761 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:12:59.710719    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:04.711414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:04.711365    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:04.711891 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:04.711477    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:13:08.788652 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:08.788601    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:13:08.789148 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:08.788596    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:13:09.710884 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:09.710838    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:14.710811 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:14.710766    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:18.788121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:18.788079    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:13:18.788532 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:18.788347    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:13:19.710936 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:19.710886    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:13:21.207300 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.207271    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:13:21.219483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.219449    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle\") pod \"58640b38-9c01-4db7-aaee-5a5b1b665374\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") "
Jul 28 16:13:21.219694 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.219502    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") pod \"58640b38-9c01-4db7-aaee-5a5b1b665374\" (UID: \"58640b38-9c01-4db7-aaee-5a5b1b665374\") "
Jul 28 16:13:21.219904 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.219878    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "58640b38-9c01-4db7-aaee-5a5b1b665374" (UID: "58640b38-9c01-4db7-aaee-5a5b1b665374"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:13:21.221798 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.221771    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58640b38-9c01-4db7-aaee-5a5b1b665374" (UID: "58640b38-9c01-4db7-aaee-5a5b1b665374"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:13:21.320222 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.320129    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58640b38-9c01-4db7-aaee-5a5b1b665374-openshift-service-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:13:21.320222 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.320159    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58640b38-9c01-4db7-aaee-5a5b1b665374-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:13:21.921890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.921858    2556 generic.go:356] "Generic (PLEG): container finished" podID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerID="f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d" exitCode=0
Jul 28 16:13:21.922071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.921907    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"
Jul 28 16:13:21.922071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.921955    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" event={"ID":"58640b38-9c01-4db7-aaee-5a5b1b665374","Type":"ContainerDied","Data":"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"}
Jul 28 16:13:21.922071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.922013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz" event={"ID":"58640b38-9c01-4db7-aaee-5a5b1b665374","Type":"ContainerDied","Data":"e3fc505fbeab9e995a4a13326198cf87ebd27eda38dfbce9d68f39d98019471b"}
Jul 28 16:13:21.922071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.922038    2556 scope.go:117] "RemoveContainer" containerID="f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"
Jul 28 16:13:21.930969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.930947    2556 scope.go:117] "RemoveContainer" containerID="f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"
Jul 28 16:13:21.931332 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:13:21.931307    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d\": container with ID starting with f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d not found: ID does not exist" containerID="f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"
Jul 28 16:13:21.931427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.931345    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d"} err="failed to get container status \"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d\": rpc error: code = NotFound desc = could not find container \"f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d\": container with ID starting with f04b5148d3b68dc593f0de37d7ebdc9e4e9c92dce74829067668ed7e4a12e97d not found: ID does not exist"
Jul 28 16:13:21.941567 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.941533    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:13:21.943691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:21.943659    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-fff6dfdd4-k75lz"]
Jul 28 16:13:23.462772 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:23.462738    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" path="/var/lib/kubelet/pods/58640b38-9c01-4db7-aaee-5a5b1b665374/volumes"
Jul 28 16:13:28.788734 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:28.788689    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:13:28.789128 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:28.788690    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:13:38.788831 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:38.788799    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:13:38.789773 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:13:38.789755    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:16:15.362239 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:16:15.362207    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:16:15.362757 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:16:15.362713    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:20:55.797887 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:55.797795    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:20:55.798407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:55.797996    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" containerID="cri-o://031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c" gracePeriod=30
Jul 28 16:20:56.122984 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.122934    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:20:56.124290 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.124260    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer"
Jul 28 16:20:56.124290 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.124291    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer"
Jul 28 16:20:56.124992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.124972    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58640b38-9c01-4db7-aaee-5a5b1b665374" containerName="model-chainer"
Jul 28 16:20:56.138154 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.138127    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.139333 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.139286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:20:56.141165 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.141133    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daf0b-kube-rbac-proxy-sar-config\""
Jul 28 16:20:56.141831 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.141808    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-daf0b-predictor-serving-cert\""
Jul 28 16:20:56.156248 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.156187    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:20:56.156521 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.156491    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" containerID="cri-o://a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d" gracePeriod=30
Jul 28 16:20:56.156653 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.156522    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kube-rbac-proxy" containerID="cri-o://6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f" gracePeriod=30
Jul 28 16:20:56.225085 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.225045    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:20:56.228959 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.228936    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.232842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.232811    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daf0b-predictor-serving-cert\""
Jul 28 16:20:56.233034 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.232963    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-daf0b-kube-rbac-proxy-sar-config\""
Jul 28 16:20:56.238247 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.238219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.238382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.238358    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.238456 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.238433    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccqf\" (UniqueName: \"kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.242749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.242722    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:20:56.276598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.276561    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:20:56.276866 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.276836    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" containerID="cri-o://7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362" gracePeriod=30
Jul 28 16:20:56.277071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.277037    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kube-rbac-proxy" containerID="cri-o://d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75" gracePeriod=30
Jul 28 16:20:56.339713 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339674    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.339907 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccqf\" (UniqueName: \"kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.339907 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339769    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.339907 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwk6\" (UniqueName: \"kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.339907 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339844    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.340115 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.339925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.340482 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.340465    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.342522 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.342500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.349226 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.349194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccqf\" (UniqueName: \"kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf\") pod \"success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") " pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.441090 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.441042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.441090 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.441092    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwk6\" (UniqueName: \"kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.441342 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.441166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.441342 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:20:56.441262    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-serving-cert: secret "error-404-isvc-daf0b-predictor-serving-cert" not found
Jul 28 16:20:56.441342 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:20:56.441338    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls podName:7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0 nodeName:}" failed. No retries permitted until 2026-07-28 16:20:56.941317678 +0000 UTC m=+1181.992121734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls") pod "error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" (UID: "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0") : secret "error-404-isvc-daf0b-predictor-serving-cert" not found
Jul 28 16:20:56.441925 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.441901    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.452524 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.452496    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwk6\" (UniqueName: \"kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.454313 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.454294    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:56.603414 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.603335    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:20:56.606209 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:20:56.606181    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd93d12a_16ab_4c47_a9bd_fdf83daf0a86.slice/crio-aa3ac9ce202750d3cf3d72e56870d0c79efb2e463ddfd0816ef44e53e40f01ea WatchSource:0}: Error finding container aa3ac9ce202750d3cf3d72e56870d0c79efb2e463ddfd0816ef44e53e40f01ea: Status 404 returned error can't find the container with id aa3ac9ce202750d3cf3d72e56870d0c79efb2e463ddfd0816ef44e53e40f01ea
Jul 28 16:20:56.946210 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.946095    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:56.949412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:56.949384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") pod \"error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") " pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:57.117366 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.117327    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerStarted","Data":"de3814ba0aada4d99cfa35e51e8d2b1ac52e2184c3deb078925750c8b0b6975f"}
Jul 28 16:20:57.117550 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.117399    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:57.117550 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.117547    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:20:57.117703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.117584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerStarted","Data":"ac3f2b5662309e0f8cbb4bd62e69119ee3a955673f486d88e33d8855106f6042"}
Jul 28 16:20:57.117703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.117613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerStarted","Data":"aa3ac9ce202750d3cf3d72e56870d0c79efb2e463ddfd0816ef44e53e40f01ea"}
Jul 28 16:20:57.119147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.119120    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:20:57.119387 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.119366    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f39f911-e715-46b1-b23e-50899a865a99" containerID="d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75" exitCode=2
Jul 28 16:20:57.119447 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.119434    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerDied","Data":"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"}
Jul 28 16:20:57.121020 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.120996    2556 generic.go:356] "Generic (PLEG): container finished" podID="5deadc31-746f-49fe-a654-d1f27457af11" containerID="6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f" exitCode=2
Jul 28 16:20:57.121110 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.121062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerDied","Data":"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"}
Jul 28 16:20:57.137094 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.137050    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podStartSLOduration=1.137032499 podStartE2EDuration="1.137032499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:57.13646892 +0000 UTC m=+1182.187272993" watchObservedRunningTime="2026-07-28 16:20:57.137032499 +0000 UTC m=+1182.187836571"
Jul 28 16:20:57.140591 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.140568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:57.289528 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:57.289496    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:20:57.292016 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:20:57.291987    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb2c05d_7e33_40f4_b59f_5fe0740fc3a0.slice/crio-0f57124f92998a2854b896bc9b9403e921d932d560852fd91bf1d4872882a02f WatchSource:0}: Error finding container 0f57124f92998a2854b896bc9b9403e921d932d560852fd91bf1d4872882a02f: Status 404 returned error can't find the container with id 0f57124f92998a2854b896bc9b9403e921d932d560852fd91bf1d4872882a02f
Jul 28 16:20:58.128046 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.128009    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerStarted","Data":"c49fbcc61ada9ad04d3271ff6848a7099b588b3529fa8e98032f0510822c79d2"}
Jul 28 16:20:58.128411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.128082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerStarted","Data":"d89c69a265f218c3e8ca5acad491e665639adeb8c2621b47e121dd7c1ea1e8c2"}
Jul 28 16:20:58.128411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.128232    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:58.128411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.128281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerStarted","Data":"0f57124f92998a2854b896bc9b9403e921d932d560852fd91bf1d4872882a02f"}
Jul 28 16:20:58.128533 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.128465    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:20:58.146388 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.146328    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podStartSLOduration=2.146303939 podStartE2EDuration="2.146303939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:20:58.1453027 +0000 UTC m=+1183.196106801" watchObservedRunningTime="2026-07-28 16:20:58.146303939 +0000 UTC m=+1183.197108012"
Jul 28 16:20:58.421693 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.421591    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 16:20:58.426071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.426038    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 16:20:58.731837 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:58.731754    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:20:59.132799 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.132767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:20:59.134488 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.134457    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:20:59.426071 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.425984    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 16:20:59.431981 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.431949    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 16:20:59.839107 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.839067    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:20:59.978490 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.978396    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") pod \"4f39f911-e715-46b1-b23e-50899a865a99\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") "
Jul 28 16:20:59.978490 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.978469    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config\") pod \"4f39f911-e715-46b1-b23e-50899a865a99\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") "
Jul 28 16:20:59.978754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.978568    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vhb\" (UniqueName: \"kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb\") pod \"4f39f911-e715-46b1-b23e-50899a865a99\" (UID: \"4f39f911-e715-46b1-b23e-50899a865a99\") "
Jul 28 16:20:59.978888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.978860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4f39f911-e715-46b1-b23e-50899a865a99" (UID: "4f39f911-e715-46b1-b23e-50899a865a99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:20:59.980760 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.980733    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f39f911-e715-46b1-b23e-50899a865a99" (UID: "4f39f911-e715-46b1-b23e-50899a865a99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:20:59.980760 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:20:59.980744    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb" (OuterVolumeSpecName: "kube-api-access-c8vhb") pod "4f39f911-e715-46b1-b23e-50899a865a99" (UID: "4f39f911-e715-46b1-b23e-50899a865a99"). InnerVolumeSpecName "kube-api-access-c8vhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:21:00.080006 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.079964    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vhb\" (UniqueName: \"kubernetes.io/projected/4f39f911-e715-46b1-b23e-50899a865a99-kube-api-access-c8vhb\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:00.080006 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.080000    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f39f911-e715-46b1-b23e-50899a865a99-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:00.080208 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.080015    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f39f911-e715-46b1-b23e-50899a865a99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:00.140013 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.139979    2556 generic.go:356] "Generic (PLEG): container finished" podID="4f39f911-e715-46b1-b23e-50899a865a99" containerID="7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362" exitCode=0
Jul 28 16:21:00.140446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.140040    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"
Jul 28 16:21:00.140446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.140072    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerDied","Data":"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"}
Jul 28 16:21:00.140446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.140166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4" event={"ID":"4f39f911-e715-46b1-b23e-50899a865a99","Type":"ContainerDied","Data":"629fc131974c52defdd2dab0e4a53cbc850840c4e249d4c8774a1ac781c6da0b"}
Jul 28 16:21:00.140446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.140196    2556 scope.go:117] "RemoveContainer" containerID="d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"
Jul 28 16:21:00.141217 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.141176    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:21:00.152266 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.152231    2556 scope.go:117] "RemoveContainer" containerID="7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"
Jul 28 16:21:00.163776 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.163743    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:21:00.168375 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.168338    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-021cc-predictor-9cd49ff5b-nhxs4"]
Jul 28 16:21:00.215803 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.215777    2556 scope.go:117] "RemoveContainer" containerID="d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"
Jul 28 16:21:00.216178 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:21:00.216155    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75\": container with ID starting with d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75 not found: ID does not exist" containerID="d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"
Jul 28 16:21:00.216227 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.216194    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75"} err="failed to get container status \"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75\": rpc error: code = NotFound desc = could not find container \"d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75\": container with ID starting with d6049b512521b65d782a73cdb1b3fb89000835c86a240a54a72163c36d6d1d75 not found: ID does not exist"
Jul 28 16:21:00.216227 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.216220    2556 scope.go:117] "RemoveContainer" containerID="7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"
Jul 28 16:21:00.216540 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:21:00.216516    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362\": container with ID starting with 7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362 not found: ID does not exist" containerID="7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"
Jul 28 16:21:00.216679 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.216545    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362"} err="failed to get container status \"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362\": rpc error: code = NotFound desc = could not find container \"7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362\": container with ID starting with 7025bc9b7896d971b637e3a769cb319bcd97c0c22231d50d1ac65f7561045362 not found: ID does not exist"
Jul 28 16:21:00.304143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.304119    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:21:00.381665 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.381604    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls\") pod \"5deadc31-746f-49fe-a654-d1f27457af11\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") "
Jul 28 16:21:00.381843 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.381702    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q57hb\" (UniqueName: \"kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb\") pod \"5deadc31-746f-49fe-a654-d1f27457af11\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") "
Jul 28 16:21:00.381843 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.381752    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config\") pod \"5deadc31-746f-49fe-a654-d1f27457af11\" (UID: \"5deadc31-746f-49fe-a654-d1f27457af11\") "
Jul 28 16:21:00.382142 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.382117    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5deadc31-746f-49fe-a654-d1f27457af11" (UID: "5deadc31-746f-49fe-a654-d1f27457af11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:21:00.383856 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.383830    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5deadc31-746f-49fe-a654-d1f27457af11" (UID: "5deadc31-746f-49fe-a654-d1f27457af11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:21:00.383964 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.383943    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb" (OuterVolumeSpecName: "kube-api-access-q57hb") pod "5deadc31-746f-49fe-a654-d1f27457af11" (UID: "5deadc31-746f-49fe-a654-d1f27457af11"). InnerVolumeSpecName "kube-api-access-q57hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:21:00.482303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.482264    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5deadc31-746f-49fe-a654-d1f27457af11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:00.482303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.482300    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5deadc31-746f-49fe-a654-d1f27457af11-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:00.482303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:00.482311    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57hb\" (UniqueName: \"kubernetes.io/projected/5deadc31-746f-49fe-a654-d1f27457af11-kube-api-access-q57hb\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:01.148274 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.148230    2556 generic.go:356] "Generic (PLEG): container finished" podID="5deadc31-746f-49fe-a654-d1f27457af11" containerID="a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d" exitCode=0
Jul 28 16:21:01.148709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.148296    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"
Jul 28 16:21:01.148709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.148312    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerDied","Data":"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"}
Jul 28 16:21:01.148709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.148382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl" event={"ID":"5deadc31-746f-49fe-a654-d1f27457af11","Type":"ContainerDied","Data":"816437aa96c887d7606a44c0cac10b9da252f7f1235c24a9234494075ec7cec4"}
Jul 28 16:21:01.148709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.148415    2556 scope.go:117] "RemoveContainer" containerID="6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"
Jul 28 16:21:01.157844 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.157826    2556 scope.go:117] "RemoveContainer" containerID="a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"
Jul 28 16:21:01.168246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.168226    2556 scope.go:117] "RemoveContainer" containerID="6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"
Jul 28 16:21:01.168605 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:21:01.168577    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f\": container with ID starting with 6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f not found: ID does not exist" containerID="6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"
Jul 28 16:21:01.168740 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.168608    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f"} err="failed to get container status \"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f\": rpc error: code = NotFound desc = could not find container \"6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f\": container with ID starting with 6a87b0a5d3e8b7f4272ff1e0589320b295da84859f5ce15433a10e0effe6877f not found: ID does not exist"
Jul 28 16:21:01.168740 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.168646    2556 scope.go:117] "RemoveContainer" containerID="a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"
Jul 28 16:21:01.169002 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:21:01.168969    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d\": container with ID starting with a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d not found: ID does not exist" containerID="a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"
Jul 28 16:21:01.169106 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.169019    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d"} err="failed to get container status \"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d\": rpc error: code = NotFound desc = could not find container \"a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d\": container with ID starting with a070334420729b0c8a2855bc4888da9cd63121d3929718f5b79300d2a709002d not found: ID does not exist"
Jul 28 16:21:01.171998 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.171970    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:21:01.175393 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.175370    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-021cc-predictor-7b688f7777-fsnkl"]
Jul 28 16:21:01.463890 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.463812    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f39f911-e715-46b1-b23e-50899a865a99" path="/var/lib/kubelet/pods/4f39f911-e715-46b1-b23e-50899a865a99/volumes"
Jul 28 16:21:01.464262 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:01.464248    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5deadc31-746f-49fe-a654-d1f27457af11" path="/var/lib/kubelet/pods/5deadc31-746f-49fe-a654-d1f27457af11/volumes"
Jul 28 16:21:03.133548 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:03.133412    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:21:03.134072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:03.133980    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:21:03.732532 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:03.732494    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:21:05.146015 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:05.145984    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:21:05.146494 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:05.146429    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:21:08.732195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:08.732158    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:21:08.732581 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:08.732257    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:21:13.134709 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:13.134669    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:21:13.732879 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:13.732840    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:21:15.146670 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:15.146603    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:21:15.392437 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:15.392404    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:21:15.393905 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:15.393880    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:21:18.731662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:18.731604    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:21:23.134979 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:23.134939    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:21:23.734680 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:23.734610    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 16:21:25.146722 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:25.146679    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:21:26.003648 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.003596    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:21:26.120600 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.120568    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle\") pod \"15a1d87c-0187-4d95-b98b-9ecd12661654\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") "
Jul 28 16:21:26.120806 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.120706    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls\") pod \"15a1d87c-0187-4d95-b98b-9ecd12661654\" (UID: \"15a1d87c-0187-4d95-b98b-9ecd12661654\") "
Jul 28 16:21:26.121038 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.121005    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "15a1d87c-0187-4d95-b98b-9ecd12661654" (UID: "15a1d87c-0187-4d95-b98b-9ecd12661654"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:21:26.122911 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.122887    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15a1d87c-0187-4d95-b98b-9ecd12661654" (UID: "15a1d87c-0187-4d95-b98b-9ecd12661654"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:21:26.221876 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.221832    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a1d87c-0187-4d95-b98b-9ecd12661654-openshift-service-ca-bundle\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:26.221876 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.221877    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15a1d87c-0187-4d95-b98b-9ecd12661654-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:21:26.280505 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.280472    2556 generic.go:356] "Generic (PLEG): container finished" podID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerID="031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c" exitCode=0
Jul 28 16:21:26.280732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.280519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" event={"ID":"15a1d87c-0187-4d95-b98b-9ecd12661654","Type":"ContainerDied","Data":"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"}
Jul 28 16:21:26.280732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.280600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n" event={"ID":"15a1d87c-0187-4d95-b98b-9ecd12661654","Type":"ContainerDied","Data":"d15a9c1102e1ef1df08aabd61efc3317f343beffa36e29a750a84dcb4a167bca"}
Jul 28 16:21:26.280732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.280650    2556 scope.go:117] "RemoveContainer" containerID="031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"
Jul 28 16:21:26.280732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.280520    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"
Jul 28 16:21:26.303821 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.303783    2556 scope.go:117] "RemoveContainer" containerID="031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"
Jul 28 16:21:26.314550 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:21:26.314511    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c\": container with ID starting with 031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c not found: ID does not exist" containerID="031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"
Jul 28 16:21:26.314721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.314575    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c"} err="failed to get container status \"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c\": rpc error: code = NotFound desc = could not find container \"031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c\": container with ID starting with 031d58d33f03f8be8cf11235c300bd1b6ec92a02a710d10d8507c29606d54e2c not found: ID does not exist"
Jul 28 16:21:26.343728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.343687    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:21:26.346381 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:26.346343    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-021cc-6486fdbd5-9fr9n"]
Jul 28 16:21:27.464529 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:27.464495    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" path="/var/lib/kubelet/pods/15a1d87c-0187-4d95-b98b-9ecd12661654/volumes"
Jul 28 16:21:33.134185 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:33.134141    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:21:35.146776 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:35.146730    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 16:21:43.134240 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:43.134193    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 16:21:45.147814 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:45.147784    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:21:53.135164 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:21:53.135133    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:22:06.050182 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.050037    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:22:06.050713 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.050346    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" containerID="cri-o://3065835da5e0295e1dc21bf791387b846bf3de3852af301db2bda4838733fa1e" gracePeriod=30
Jul 28 16:22:06.050788 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.050758    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kube-rbac-proxy" containerID="cri-o://f6ae9ba6dba11fb73c7bc3001d58f2f0975cf9e9792a8a388a3115e1d06b37ae" gracePeriod=30
Jul 28 16:22:06.131371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131339    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:22:06.131804 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131790    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc"
Jul 28 16:22:06.131864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131806    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc"
Jul 28 16:22:06.131864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131820    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kube-rbac-proxy"
Jul 28 16:22:06.131864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131825    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kube-rbac-proxy"
Jul 28 16:22:06.131864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131831    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kube-rbac-proxy"
Jul 28 16:22:06.131864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131838    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kube-rbac-proxy"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131875    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131881    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131889    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.131894    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.132012    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kube-rbac-proxy"
Jul 28 16:22:06.132036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.132023    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a1d87c-0187-4d95-b98b-9ecd12661654" containerName="switch-graph-021cc"
Jul 28 16:22:06.132237 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.132046    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kube-rbac-proxy"
Jul 28 16:22:06.132237 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.132057    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5deadc31-746f-49fe-a654-d1f27457af11" containerName="kserve-container"
Jul 28 16:22:06.132237 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.132064    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f39f911-e715-46b1-b23e-50899a865a99" containerName="kserve-container"
Jul 28 16:22:06.135921 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.135884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.138869 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.138841    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c1b1-predictor-serving-cert\""
Jul 28 16:22:06.139191 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.139164    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7c1b1-kube-rbac-proxy-sar-config\""
Jul 28 16:22:06.145043 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.145017    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:22:06.145272 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.145249    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" containerID="cri-o://a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c" gracePeriod=30
Jul 28 16:22:06.145384 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.145354    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kube-rbac-proxy" containerID="cri-o://3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8" gracePeriod=30
Jul 28 16:22:06.153726 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.153428    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:22:06.178981 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.178953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.179136 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.179000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.179136 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.179037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595t6\" (UniqueName: \"kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.279555 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.279517    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.279787 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.279570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.279787 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.279607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595t6\" (UniqueName: \"kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.279787 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:06.279712    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-serving-cert: secret "success-200-isvc-7c1b1-predictor-serving-cert" not found
Jul 28 16:22:06.279963 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:06.279795    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls podName:f8e0ce44-1af6-49d1-80f6-2edca34b926a nodeName:}" failed. No retries permitted until 2026-07-28 16:22:06.77977424 +0000 UTC m=+1251.830578312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls") pod "success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" (UID: "f8e0ce44-1af6-49d1-80f6-2edca34b926a") : secret "success-200-isvc-7c1b1-predictor-serving-cert" not found
Jul 28 16:22:06.280315 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.280297    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.285602 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.285572    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:22:06.289305 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.289282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.290882 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.290862    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595t6\" (UniqueName: \"kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.291930 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.291888    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c1b1-kube-rbac-proxy-sar-config\""
Jul 28 16:22:06.292138 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.292118    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7c1b1-predictor-serving-cert\""
Jul 28 16:22:06.301554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.301482    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:22:06.380497 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.380454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.380728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.380510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.380728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.380692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msc7b\" (UniqueName: \"kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.478504 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.478471    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerID="f6ae9ba6dba11fb73c7bc3001d58f2f0975cf9e9792a8a388a3115e1d06b37ae" exitCode=2
Jul 28 16:22:06.478719 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.478549    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerDied","Data":"f6ae9ba6dba11fb73c7bc3001d58f2f0975cf9e9792a8a388a3115e1d06b37ae"}
Jul 28 16:22:06.480350 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.480318    2556 generic.go:356] "Generic (PLEG): container finished" podID="675be705-9def-42cb-983c-5a12e49fe04a" containerID="3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8" exitCode=2
Jul 28 16:22:06.480483 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.480386    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerDied","Data":"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"}
Jul 28 16:22:06.481821 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.481800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.481909 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.481829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.481909 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.481879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msc7b\" (UniqueName: \"kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.482451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.482435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.484733 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.484711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.493235 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.493210    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msc7b\" (UniqueName: \"kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b\") pod \"error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") " pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.610553 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.610509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:06.756309 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.756282    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:22:06.759003 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:22:06.758972    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3534c90a_ca82_4e2c_8964_cd2439ba2fee.slice/crio-c532710a04484d932a1818e4520a5c8666d83a1f44545a0f41a4dd73b297e8fa WatchSource:0}: Error finding container c532710a04484d932a1818e4520a5c8666d83a1f44545a0f41a4dd73b297e8fa: Status 404 returned error can't find the container with id c532710a04484d932a1818e4520a5c8666d83a1f44545a0f41a4dd73b297e8fa
Jul 28 16:22:06.783922 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.783892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:06.786544 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:06.786508    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") pod \"success-200-isvc-7c1b1-predictor-5f679bf9f-px27h\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") " pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:07.049611 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.049564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:07.192512 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.192475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:22:07.195986 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:22:07.195953    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e0ce44_1af6_49d1_80f6_2edca34b926a.slice/crio-ca3059b3022fb715a83a1c634c0166fa6a8f04bb39a4de335c73c25aa795a5ac WatchSource:0}: Error finding container ca3059b3022fb715a83a1c634c0166fa6a8f04bb39a4de335c73c25aa795a5ac: Status 404 returned error can't find the container with id ca3059b3022fb715a83a1c634c0166fa6a8f04bb39a4de335c73c25aa795a5ac
Jul 28 16:22:07.487550 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.487450    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerStarted","Data":"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"}
Jul 28 16:22:07.487550 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.487520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerStarted","Data":"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"}
Jul 28 16:22:07.487772 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.487554    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:07.487772 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.487583    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:07.487772 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.487604    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerStarted","Data":"c532710a04484d932a1818e4520a5c8666d83a1f44545a0f41a4dd73b297e8fa"}
Jul 28 16:22:07.489288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.489254    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:07.489542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.489518    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerStarted","Data":"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"}
Jul 28 16:22:07.489597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.489570    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerStarted","Data":"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"}
Jul 28 16:22:07.489597 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.489589    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerStarted","Data":"ca3059b3022fb715a83a1c634c0166fa6a8f04bb39a4de335c73c25aa795a5ac"}
Jul 28 16:22:07.489724 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.489612    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:07.513853 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.513800    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podStartSLOduration=1.513782052 podStartE2EDuration="1.513782052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:07.512471316 +0000 UTC m=+1252.563275390" watchObservedRunningTime="2026-07-28 16:22:07.513782052 +0000 UTC m=+1252.564586125"
Jul 28 16:22:07.535389 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:07.535332    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podStartSLOduration=1.535314385 podStartE2EDuration="1.535314385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:07.532973343 +0000 UTC m=+1252.583777415" watchObservedRunningTime="2026-07-28 16:22:07.535314385 +0000 UTC m=+1252.586118457"
Jul 28 16:22:08.494731 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.494697    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:08.495132 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.494763    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:08.496063 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.496039    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:08.783669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.783542    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 16:22:08.783669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.783553    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 16:22:08.788921 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.788889    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 16:22:08.789017 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:08.788889    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 16:22:09.501485 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.501455    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerID="3065835da5e0295e1dc21bf791387b846bf3de3852af301db2bda4838733fa1e" exitCode=0
Jul 28 16:22:09.501937 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.501526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerDied","Data":"3065835da5e0295e1dc21bf791387b846bf3de3852af301db2bda4838733fa1e"}
Jul 28 16:22:09.501937 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.501582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q" event={"ID":"e1de457f-a67f-40b2-98de-c159c23a9a0a","Type":"ContainerDied","Data":"4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc"}
Jul 28 16:22:09.501937 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.501602    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4280116ebc89b9990f3c29cbda52a86247230fea772d01a5cf60c10fad6113cc"
Jul 28 16:22:09.501937 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.501883    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:09.515907 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.515879    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:22:09.607350 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.607251    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config\") pod \"e1de457f-a67f-40b2-98de-c159c23a9a0a\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") "
Jul 28 16:22:09.607350 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.607290    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls\") pod \"e1de457f-a67f-40b2-98de-c159c23a9a0a\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") "
Jul 28 16:22:09.607350 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.607323    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n\") pod \"e1de457f-a67f-40b2-98de-c159c23a9a0a\" (UID: \"e1de457f-a67f-40b2-98de-c159c23a9a0a\") "
Jul 28 16:22:09.607865 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.607831    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1de457f-a67f-40b2-98de-c159c23a9a0a" (UID: "e1de457f-a67f-40b2-98de-c159c23a9a0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:09.609972 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.609946    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1de457f-a67f-40b2-98de-c159c23a9a0a" (UID: "e1de457f-a67f-40b2-98de-c159c23a9a0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:09.610210 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.610185    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n" (OuterVolumeSpecName: "kube-api-access-rbw6n") pod "e1de457f-a67f-40b2-98de-c159c23a9a0a" (UID: "e1de457f-a67f-40b2-98de-c159c23a9a0a"). InnerVolumeSpecName "kube-api-access-rbw6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:09.703812 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.703785    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:22:09.708328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.708302    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:09.708328 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.708327    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1de457f-a67f-40b2-98de-c159c23a9a0a-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:09.708511 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.708337    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbw6n\" (UniqueName: \"kubernetes.io/projected/e1de457f-a67f-40b2-98de-c159c23a9a0a-kube-api-access-rbw6n\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:09.809537 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.809492    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khtds\" (UniqueName: \"kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds\") pod \"675be705-9def-42cb-983c-5a12e49fe04a\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") "
Jul 28 16:22:09.809767 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.809569    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls\") pod \"675be705-9def-42cb-983c-5a12e49fe04a\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") "
Jul 28 16:22:09.809767 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.809675    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config\") pod \"675be705-9def-42cb-983c-5a12e49fe04a\" (UID: \"675be705-9def-42cb-983c-5a12e49fe04a\") "
Jul 28 16:22:09.810212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.809990    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "675be705-9def-42cb-983c-5a12e49fe04a" (UID: "675be705-9def-42cb-983c-5a12e49fe04a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:09.811885 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.811864    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds" (OuterVolumeSpecName: "kube-api-access-khtds") pod "675be705-9def-42cb-983c-5a12e49fe04a" (UID: "675be705-9def-42cb-983c-5a12e49fe04a"). InnerVolumeSpecName "kube-api-access-khtds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:09.811955 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.811905    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "675be705-9def-42cb-983c-5a12e49fe04a" (UID: "675be705-9def-42cb-983c-5a12e49fe04a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:09.910513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.910424    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/675be705-9def-42cb-983c-5a12e49fe04a-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:09.910513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.910461    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/675be705-9def-42cb-983c-5a12e49fe04a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:09.910513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:09.910473    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khtds\" (UniqueName: \"kubernetes.io/projected/675be705-9def-42cb-983c-5a12e49fe04a-kube-api-access-khtds\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:10.508058 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508024    2556 generic.go:356] "Generic (PLEG): container finished" podID="675be705-9def-42cb-983c-5a12e49fe04a" containerID="a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c" exitCode=0
Jul 28 16:22:10.508523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508077    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"
Jul 28 16:22:10.508523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerDied","Data":"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"}
Jul 28 16:22:10.508523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb" event={"ID":"675be705-9def-42cb-983c-5a12e49fe04a","Type":"ContainerDied","Data":"36d8b14b86d447c4f2a84c8e5d2429991215694f8696e6a050282ebe6173db17"}
Jul 28 16:22:10.508523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508224    2556 scope.go:117] "RemoveContainer" containerID="3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"
Jul 28 16:22:10.508523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.508346    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"
Jul 28 16:22:10.518604 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.518582    2556 scope.go:117] "RemoveContainer" containerID="a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"
Jul 28 16:22:10.527320 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.527301    2556 scope.go:117] "RemoveContainer" containerID="3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"
Jul 28 16:22:10.527602 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:10.527582    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8\": container with ID starting with 3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8 not found: ID does not exist" containerID="3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"
Jul 28 16:22:10.527794 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.527610    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8"} err="failed to get container status \"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8\": rpc error: code = NotFound desc = could not find container \"3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8\": container with ID starting with 3b2adc15c2eb69e0290a20fedac7a7a1aad6fa6c4677043a7658f55380a5fad8 not found: ID does not exist"
Jul 28 16:22:10.527794 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.527647    2556 scope.go:117] "RemoveContainer" containerID="a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"
Jul 28 16:22:10.527906 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:10.527889    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c\": container with ID starting with a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c not found: ID does not exist" containerID="a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"
Jul 28 16:22:10.527991 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.527913    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c"} err="failed to get container status \"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c\": rpc error: code = NotFound desc = could not find container \"a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c\": container with ID starting with a1c3e673b201ff77f1d8d2f0ae13aa2f1ad3c43e9737b58f54dcb0a1b0fdb60c not found: ID does not exist"
Jul 28 16:22:10.537216 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.537183    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:22:10.543360 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.543331    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0402e-predictor-74b6c76776-g66wb"]
Jul 28 16:22:10.560739 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.560703    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:22:10.565776 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:10.565739    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0402e-predictor-5f7c85c4d9-m8d6q"]
Jul 28 16:22:11.463453 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:11.463416    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675be705-9def-42cb-983c-5a12e49fe04a" path="/var/lib/kubelet/pods/675be705-9def-42cb-983c-5a12e49fe04a/volumes"
Jul 28 16:22:11.465725 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:11.463885    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" path="/var/lib/kubelet/pods/e1de457f-a67f-40b2-98de-c159c23a9a0a/volumes"
Jul 28 16:22:13.498787 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:13.498761    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:22:13.499260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:13.499235    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:14.506612 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:14.506580    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:22:14.507218 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:14.507194    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:16.327178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.327139    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:22:16.327613 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.327384    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container" containerID="cri-o://ac3f2b5662309e0f8cbb4bd62e69119ee3a955673f486d88e33d8855106f6042" gracePeriod=30
Jul 28 16:22:16.327613 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.327411    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kube-rbac-proxy" containerID="cri-o://de3814ba0aada4d99cfa35e51e8d2b1ac52e2184c3deb078925750c8b0b6975f" gracePeriod=30
Jul 28 16:22:16.392735 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.392696    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:22:16.393189 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393167    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393190    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393229    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393238    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393250    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393259    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393272    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container"
Jul 28 16:22:16.393289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393281    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container"
Jul 28 16:22:16.393669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393371    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kserve-container"
Jul 28 16:22:16.393669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393383    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="675be705-9def-42cb-983c-5a12e49fe04a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.393669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393394    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kserve-container"
Jul 28 16:22:16.393669 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.393407    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1de457f-a67f-40b2-98de-c159c23a9a0a" containerName="kube-rbac-proxy"
Jul 28 16:22:16.397036 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.397014    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:22:16.397170 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.397136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.397220 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.397198    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container" containerID="cri-o://d89c69a265f218c3e8ca5acad491e665639adeb8c2621b47e121dd7c1ea1e8c2" gracePeriod=30
Jul 28 16:22:16.397262 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.397228    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kube-rbac-proxy" containerID="cri-o://c49fbcc61ada9ad04d3271ff6848a7099b588b3529fa8e98032f0510822c79d2" gracePeriod=30
Jul 28 16:22:16.399659 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.399609    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e5aa9-predictor-serving-cert\""
Jul 28 16:22:16.399963 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.399932    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e5aa9-kube-rbac-proxy-sar-config\""
Jul 28 16:22:16.409111 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.408979    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:22:16.471550 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.471521    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.471728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.471560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjz8\" (UniqueName: \"kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.471728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.471690    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.505767 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.505731    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:22:16.509286 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.509261    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.512007 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.511982    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e5aa9-kube-rbac-proxy-sar-config\""
Jul 28 16:22:16.512198 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.512039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e5aa9-predictor-serving-cert\""
Jul 28 16:22:16.520842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.520229    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:22:16.559282 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.559246    2556 generic.go:356] "Generic (PLEG): container finished" podID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerID="c49fbcc61ada9ad04d3271ff6848a7099b588b3529fa8e98032f0510822c79d2" exitCode=2
Jul 28 16:22:16.559466 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.559324    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerDied","Data":"c49fbcc61ada9ad04d3271ff6848a7099b588b3529fa8e98032f0510822c79d2"}
Jul 28 16:22:16.561198 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.561176    2556 generic.go:356] "Generic (PLEG): container finished" podID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerID="de3814ba0aada4d99cfa35e51e8d2b1ac52e2184c3deb078925750c8b0b6975f" exitCode=2
Jul 28 16:22:16.561319 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.561206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerDied","Data":"de3814ba0aada4d99cfa35e51e8d2b1ac52e2184c3deb078925750c8b0b6975f"}
Jul 28 16:22:16.572222 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc59p\" (UniqueName: \"kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.572364 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.572364 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.572462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.572462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjz8\" (UniqueName: \"kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.572462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.572442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.572582 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:16.572505    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-serving-cert: secret "success-200-isvc-e5aa9-predictor-serving-cert" not found
Jul 28 16:22:16.572673 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:16.572579    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls podName:af2e7a73-8319-4118-a799-4d8a2fdfd4e7 nodeName:}" failed. No retries permitted until 2026-07-28 16:22:17.072557159 +0000 UTC m=+1262.123361217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls") pod "success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" (UID: "af2e7a73-8319-4118-a799-4d8a2fdfd4e7") : secret "success-200-isvc-e5aa9-predictor-serving-cert" not found
Jul 28 16:22:16.573067 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.573048    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.581279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.581218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjz8\" (UniqueName: \"kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:16.673335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.673301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.673335 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.673351    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc59p\" (UniqueName: \"kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.673569 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.673404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.673569 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:16.673463    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-serving-cert: secret "error-404-isvc-e5aa9-predictor-serving-cert" not found
Jul 28 16:22:16.673569 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:22:16.673530    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls podName:9cec1195-36c6-4155-838e-f16d928c184b nodeName:}" failed. No retries permitted until 2026-07-28 16:22:17.173512287 +0000 UTC m=+1262.224316339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls") pod "error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" (UID: "9cec1195-36c6-4155-838e-f16d928c184b") : secret "error-404-isvc-e5aa9-predictor-serving-cert" not found
Jul 28 16:22:16.674088 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.674071    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:16.683229 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:16.683199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc59p\" (UniqueName: \"kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:17.077303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.077254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:17.079860 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.079836    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") pod \"success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") " pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:17.178673 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.178601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:17.181150 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.181130    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") pod \"error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") " pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:17.285916 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.285885    2556 scope.go:117] "RemoveContainer" containerID="f6ae9ba6dba11fb73c7bc3001d58f2f0975cf9e9792a8a388a3115e1d06b37ae"
Jul 28 16:22:17.297808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.297786    2556 scope.go:117] "RemoveContainer" containerID="3065835da5e0295e1dc21bf791387b846bf3de3852af301db2bda4838733fa1e"
Jul 28 16:22:17.317513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.317492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:17.421938 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.421897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:17.463115 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:22:17.463080    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2e7a73_8319_4118_a799_4d8a2fdfd4e7.slice/crio-ac3a67082dbf47eecde3837bf5e72ca51481f78505ce73d5e2097b9deb1511e0 WatchSource:0}: Error finding container ac3a67082dbf47eecde3837bf5e72ca51481f78505ce73d5e2097b9deb1511e0: Status 404 returned error can't find the container with id ac3a67082dbf47eecde3837bf5e72ca51481f78505ce73d5e2097b9deb1511e0
Jul 28 16:22:17.465081 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.465055    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:22:17.568488 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.568461    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerStarted","Data":"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"}
Jul 28 16:22:17.568601 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.568518    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerStarted","Data":"ac3a67082dbf47eecde3837bf5e72ca51481f78505ce73d5e2097b9deb1511e0"}
Jul 28 16:22:17.576204 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:17.576178    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:22:17.578887 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:22:17.578854    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cec1195_36c6_4155_838e_f16d928c184b.slice/crio-5120de93a0a5401d75007218f2ed882553d5aae91db0f3258a1d03dd4f7ca133 WatchSource:0}: Error finding container 5120de93a0a5401d75007218f2ed882553d5aae91db0f3258a1d03dd4f7ca133: Status 404 returned error can't find the container with id 5120de93a0a5401d75007218f2ed882553d5aae91db0f3258a1d03dd4f7ca133
Jul 28 16:22:18.128795 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.128744    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 16:22:18.574976 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.574872    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerStarted","Data":"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"}
Jul 28 16:22:18.575388 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.575048    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:18.575388 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.575093    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:18.576415 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.576386    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:18.576788 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.576765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerStarted","Data":"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"}
Jul 28 16:22:18.576857 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.576821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerStarted","Data":"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"}
Jul 28 16:22:18.576896 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.576861    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerStarted","Data":"5120de93a0a5401d75007218f2ed882553d5aae91db0f3258a1d03dd4f7ca133"}
Jul 28 16:22:18.576928 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.576894    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:18.600703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.600644    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podStartSLOduration=2.600609657 podStartE2EDuration="2.600609657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:18.598300206 +0000 UTC m=+1263.649104280" watchObservedRunningTime="2026-07-28 16:22:18.600609657 +0000 UTC m=+1263.651413761"
Jul 28 16:22:18.619926 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:18.619874    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podStartSLOduration=2.619858014 podStartE2EDuration="2.619858014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:22:18.617080132 +0000 UTC m=+1263.667884206" watchObservedRunningTime="2026-07-28 16:22:18.619858014 +0000 UTC m=+1263.670662086"
Jul 28 16:22:19.582060 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:19.582021    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:19.582542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:19.582072    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:19.583245 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:19.583222    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:22:20.141059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.141011    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 16:22:20.588860 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.588827    2556 generic.go:356] "Generic (PLEG): container finished" podID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerID="ac3f2b5662309e0f8cbb4bd62e69119ee3a955673f486d88e33d8855106f6042" exitCode=0
Jul 28 16:22:20.589279 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.588905    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerDied","Data":"ac3f2b5662309e0f8cbb4bd62e69119ee3a955673f486d88e33d8855106f6042"}
Jul 28 16:22:20.590866 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.590842    2556 generic.go:356] "Generic (PLEG): container finished" podID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerID="d89c69a265f218c3e8ca5acad491e665639adeb8c2621b47e121dd7c1ea1e8c2" exitCode=0
Jul 28 16:22:20.590986 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.590912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerDied","Data":"d89c69a265f218c3e8ca5acad491e665639adeb8c2621b47e121dd7c1ea1e8c2"}
Jul 28 16:22:20.591256 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.591230    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:22:20.686246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.686219    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:22:20.813107 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.813074    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gccqf\" (UniqueName: \"kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf\") pod \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") "
Jul 28 16:22:20.813288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.813131    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls\") pod \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") "
Jul 28 16:22:20.813288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.813160    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config\") pod \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\" (UID: \"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86\") "
Jul 28 16:22:20.813675 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.813645    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" (UID: "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:20.816692 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.816653    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" (UID: "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:20.816692 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.816664    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf" (OuterVolumeSpecName: "kube-api-access-gccqf") pod "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" (UID: "bd93d12a-16ab-4c47-a9bd-fdf83daf0a86"). InnerVolumeSpecName "kube-api-access-gccqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:20.842180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.842156    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:22:20.913844 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.913815    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") pod \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") "
Jul 28 16:22:20.914011 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.913872    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwk6\" (UniqueName: \"kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6\") pod \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") "
Jul 28 16:22:20.914011 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.913946    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config\") pod \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\" (UID: \"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0\") "
Jul 28 16:22:20.914239 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.914218    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccqf\" (UniqueName: \"kubernetes.io/projected/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-api-access-gccqf\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:20.914317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.914244    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:20.914317 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.914259    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:20.914416 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.914353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" (UID: "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:22:20.916310 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.916274    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" (UID: "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:22:20.916427 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:20.916312    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6" (OuterVolumeSpecName: "kube-api-access-plwk6") pod "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" (UID: "7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0"). InnerVolumeSpecName "kube-api-access-plwk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:22:21.014958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.014858    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:21.014958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.014902    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwk6\" (UniqueName: \"kubernetes.io/projected/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-api-access-plwk6\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:21.014958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.014912    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:22:21.597917 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.597881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn" event={"ID":"bd93d12a-16ab-4c47-a9bd-fdf83daf0a86","Type":"ContainerDied","Data":"aa3ac9ce202750d3cf3d72e56870d0c79efb2e463ddfd0816ef44e53e40f01ea"}
Jul 28 16:22:21.598352 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.597893    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"
Jul 28 16:22:21.598352 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.597955    2556 scope.go:117] "RemoveContainer" containerID="de3814ba0aada4d99cfa35e51e8d2b1ac52e2184c3deb078925750c8b0b6975f"
Jul 28 16:22:21.599657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.599614    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"
Jul 28 16:22:21.599657 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.599649    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv" event={"ID":"7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0","Type":"ContainerDied","Data":"0f57124f92998a2854b896bc9b9403e921d932d560852fd91bf1d4872882a02f"}
Jul 28 16:22:21.607257 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.607221    2556 scope.go:117] "RemoveContainer" containerID="ac3f2b5662309e0f8cbb4bd62e69119ee3a955673f486d88e33d8855106f6042"
Jul 28 16:22:21.616201 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.616174    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:22:21.616295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.616187    2556 scope.go:117] "RemoveContainer" containerID="c49fbcc61ada9ad04d3271ff6848a7099b588b3529fa8e98032f0510822c79d2"
Jul 28 16:22:21.620319 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.620291    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-daf0b-predictor-788fff4fdd-nkfxv"]
Jul 28 16:22:21.625330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.625313    2556 scope.go:117] "RemoveContainer" containerID="d89c69a265f218c3e8ca5acad491e665639adeb8c2621b47e121dd7c1ea1e8c2"
Jul 28 16:22:21.631701 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.631676    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:22:21.637895 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:21.637867    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-daf0b-predictor-5b75479fb9-k2bzn"]
Jul 28 16:22:23.463487 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:23.463456    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" path="/var/lib/kubelet/pods/7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0/volumes"
Jul 28 16:22:23.463925 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:23.463910    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" path="/var/lib/kubelet/pods/bd93d12a-16ab-4c47-a9bd-fdf83daf0a86/volumes"
Jul 28 16:22:23.499412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:23.499376    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:24.507194 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:24.507160    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:24.586270 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:24.586240    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:22:24.586658 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:24.586604    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:25.596085 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:25.596054    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:22:25.596716 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:25.596690    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:22:33.499774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:33.499732    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:34.508049 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:34.508007    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:34.586869 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:34.586828    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:35.596882 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:35.596839    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:22:43.500010 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:43.499969    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:44.507682 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:44.507643    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:44.587083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:44.587047    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:45.597195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:45.597160    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:22:53.499325 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:53.499285    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 16:22:54.507661 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:54.507601    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 16:22:54.586722 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:54.586684    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:22:55.597134 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:22:55.597089    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:23:03.500326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:03.500293    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:23:04.508370 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:04.508342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:23:04.587004 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:04.586966    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:23:05.597432 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:05.597387    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:23:14.588308 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:14.588273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:23:15.598687 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:15.598654    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:23:26.309717 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.309669    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:23:26.310144 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.309975    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container" containerID="cri-o://fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4" gracePeriod=30
Jul 28 16:23:26.310144 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.310026    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kube-rbac-proxy" containerID="cri-o://209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9" gracePeriod=30
Jul 28 16:23:26.372969 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.372932    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:23:26.373263 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.373227    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container" containerID="cri-o://e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a" gracePeriod=30
Jul 28 16:23:26.373401 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.373321    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kube-rbac-proxy" containerID="cri-o://99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5" gracePeriod=30
Jul 28 16:23:26.456637 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.456585    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:23:26.457096 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457077    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457098    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457111    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457118    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457127    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457135    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457164    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457171    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457259    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457273    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kserve-container"
Jul 28 16:23:26.457538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457288    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eb2c05d-7e33-40f4-b59f-5fe0740fc3a0" containerName="kube-rbac-proxy"
Jul 28 16:23:26.457538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.457301    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd93d12a-16ab-4c47-a9bd-fdf83daf0a86" containerName="kserve-container"
Jul 28 16:23:26.460864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.460831    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.464373 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.464343    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3dc2-predictor-serving-cert\""
Jul 28 16:23:26.464769 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.464748    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c3dc2-kube-rbac-proxy-sar-config\""
Jul 28 16:23:26.476711 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.476677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:23:26.567841 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.567734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bg9\" (UniqueName: \"kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.568038 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.567960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.568102 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.568069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.646397 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.646359    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:23:26.650772 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.650748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.669063 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.669023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.669063 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.669063    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bg9\" (UniqueName: \"kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.669322 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.669118    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.669793 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.669774    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.671559 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.671528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.673985 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.673967    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3dc2-predictor-serving-cert\""
Jul 28 16:23:26.674251 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.674238    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3dc2-kube-rbac-proxy-sar-config\""
Jul 28 16:23:26.696897 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.696863    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:23:26.697172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.697156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bg9\" (UniqueName: \"kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9\") pod \"success-200-isvc-c3dc2-predictor-58b487689c-z5qjm\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") " pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.770282 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.770243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.770452 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.770313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrfd\" (UniqueName: \"kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.770452 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.770358    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.772125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.772088    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:26.871312 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.871279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.871490 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.871386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrfd\" (UniqueName: \"kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.871490 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.871454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.872271 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.872245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.874000 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.873978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.882989 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.882923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrfd\" (UniqueName: \"kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd\") pod \"error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") " pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:26.921825 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.921803    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:23:26.924659 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:23:26.924616    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3369701d_a56c_44c5_bff1_8ecd97555590.slice/crio-bfc6fa1cc5432139debec51bd0bab678ba328ff65e9c15b973cf91284b994d2f WatchSource:0}: Error finding container bfc6fa1cc5432139debec51bd0bab678ba328ff65e9c15b973cf91284b994d2f: Status 404 returned error can't find the container with id bfc6fa1cc5432139debec51bd0bab678ba328ff65e9c15b973cf91284b994d2f
Jul 28 16:23:26.932471 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.932443    2556 generic.go:356] "Generic (PLEG): container finished" podID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerID="209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9" exitCode=2
Jul 28 16:23:26.932561 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.932515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerDied","Data":"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"}
Jul 28 16:23:26.933841 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.933817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerStarted","Data":"bfc6fa1cc5432139debec51bd0bab678ba328ff65e9c15b973cf91284b994d2f"}
Jul 28 16:23:26.935641 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.935600    2556 generic.go:356] "Generic (PLEG): container finished" podID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerID="99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5" exitCode=2
Jul 28 16:23:26.935733 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.935647    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerDied","Data":"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"}
Jul 28 16:23:26.962830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:26.962566    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:27.125999 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.125966    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:23:27.129391 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:23:27.129362    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec669095_1542_45fb_bfc9_cf8ca7b37332.slice/crio-dbdfa250d0b545e287170c9e7683681edbeb33a0bd7e22cfd73aef35c5619955 WatchSource:0}: Error finding container dbdfa250d0b545e287170c9e7683681edbeb33a0bd7e22cfd73aef35c5619955: Status 404 returned error can't find the container with id dbdfa250d0b545e287170c9e7683681edbeb33a0bd7e22cfd73aef35c5619955
Jul 28 16:23:27.942721 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.942677    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerStarted","Data":"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"}
Jul 28 16:23:27.943203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.942781    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:27.943203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.942828    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerStarted","Data":"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"}
Jul 28 16:23:27.943203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.943018    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:27.944137 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.944104    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:23:27.945243 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.945217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerStarted","Data":"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"}
Jul 28 16:23:27.945371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.945272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerStarted","Data":"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"}
Jul 28 16:23:27.945371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.945296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerStarted","Data":"dbdfa250d0b545e287170c9e7683681edbeb33a0bd7e22cfd73aef35c5619955"}
Jul 28 16:23:27.945371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.945320    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:27.962584 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.962536    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podStartSLOduration=1.962519499 podStartE2EDuration="1.962519499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:27.962351547 +0000 UTC m=+1333.013155633" watchObservedRunningTime="2026-07-28 16:23:27.962519499 +0000 UTC m=+1333.013323572"
Jul 28 16:23:27.984330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:27.984276    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podStartSLOduration=1.984258308 podStartE2EDuration="1.984258308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:23:27.982833645 +0000 UTC m=+1333.033637731" watchObservedRunningTime="2026-07-28 16:23:27.984258308 +0000 UTC m=+1333.035062383"
Jul 28 16:23:28.495429 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:28.495383    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 16:23:28.952535 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:28.952489    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:28.952951 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:28.952501    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:23:28.954029 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:28.954003    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:23:29.502476 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:29.502433    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 16:23:29.958287 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:29.958243    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:23:30.193751 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.193724    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:23:30.196767 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.196715    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:23:30.307371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307329    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config\") pod \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") "
Jul 28 16:23:30.307575 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307411    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") pod \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") "
Jul 28 16:23:30.307575 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307437    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config\") pod \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") "
Jul 28 16:23:30.307575 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307555    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls\") pod \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") "
Jul 28 16:23:30.307790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307608    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msc7b\" (UniqueName: \"kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b\") pod \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\" (UID: \"3534c90a-ca82-4e2c-8964-cd2439ba2fee\") "
Jul 28 16:23:30.307790 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307711    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-595t6\" (UniqueName: \"kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6\") pod \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\" (UID: \"f8e0ce44-1af6-49d1-80f6-2edca34b926a\") "
Jul 28 16:23:30.307898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307802    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3534c90a-ca82-4e2c-8964-cd2439ba2fee" (UID: "3534c90a-ca82-4e2c-8964-cd2439ba2fee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:30.307898 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.307838    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8e0ce44-1af6-49d1-80f6-2edca34b926a" (UID: "f8e0ce44-1af6-49d1-80f6-2edca34b926a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:23:30.308066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.308032    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.308066 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.308057    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.309899 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.309875    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b" (OuterVolumeSpecName: "kube-api-access-msc7b") pod "3534c90a-ca82-4e2c-8964-cd2439ba2fee" (UID: "3534c90a-ca82-4e2c-8964-cd2439ba2fee"). InnerVolumeSpecName "kube-api-access-msc7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:23:30.309993 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.309919    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8e0ce44-1af6-49d1-80f6-2edca34b926a" (UID: "f8e0ce44-1af6-49d1-80f6-2edca34b926a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:23:30.310356 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.310334    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3534c90a-ca82-4e2c-8964-cd2439ba2fee" (UID: "3534c90a-ca82-4e2c-8964-cd2439ba2fee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:23:30.310467 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.310355    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6" (OuterVolumeSpecName: "kube-api-access-595t6") pod "f8e0ce44-1af6-49d1-80f6-2edca34b926a" (UID: "f8e0ce44-1af6-49d1-80f6-2edca34b926a"). InnerVolumeSpecName "kube-api-access-595t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:23:30.408816 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.408779    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e0ce44-1af6-49d1-80f6-2edca34b926a-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.408816 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.408809    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3534c90a-ca82-4e2c-8964-cd2439ba2fee-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.408816 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.408819    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msc7b\" (UniqueName: \"kubernetes.io/projected/3534c90a-ca82-4e2c-8964-cd2439ba2fee-kube-api-access-msc7b\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.409057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.408829    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595t6\" (UniqueName: \"kubernetes.io/projected/f8e0ce44-1af6-49d1-80f6-2edca34b926a-kube-api-access-595t6\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:23:30.965256 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.965226    2556 generic.go:356] "Generic (PLEG): container finished" podID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerID="e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a" exitCode=0
Jul 28 16:23:30.965749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.965334    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerDied","Data":"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"}
Jul 28 16:23:30.965749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.965349    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"
Jul 28 16:23:30.965749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.965402    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h" event={"ID":"f8e0ce44-1af6-49d1-80f6-2edca34b926a","Type":"ContainerDied","Data":"ca3059b3022fb715a83a1c634c0166fa6a8f04bb39a4de335c73c25aa795a5ac"}
Jul 28 16:23:30.965749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.965441    2556 scope.go:117] "RemoveContainer" containerID="99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"
Jul 28 16:23:30.967147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.967115    2556 generic.go:356] "Generic (PLEG): container finished" podID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerID="fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4" exitCode=0
Jul 28 16:23:30.967324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.967238    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"
Jul 28 16:23:30.967324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.967241    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerDied","Data":"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"}
Jul 28 16:23:30.967324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.967300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb" event={"ID":"3534c90a-ca82-4e2c-8964-cd2439ba2fee","Type":"ContainerDied","Data":"c532710a04484d932a1818e4520a5c8666d83a1f44545a0f41a4dd73b297e8fa"}
Jul 28 16:23:30.975665 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.975614    2556 scope.go:117] "RemoveContainer" containerID="e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"
Jul 28 16:23:30.984306 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.984286    2556 scope.go:117] "RemoveContainer" containerID="99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"
Jul 28 16:23:30.984607 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:23:30.984588    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5\": container with ID starting with 99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5 not found: ID does not exist" containerID="99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"
Jul 28 16:23:30.984716 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.984615    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5"} err="failed to get container status \"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5\": rpc error: code = NotFound desc = could not find container \"99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5\": container with ID starting with 99c0d16f4d6e0cb379904c265c2b9d860bcc41ab2c332991e8bed271878f89c5 not found: ID does not exist"
Jul 28 16:23:30.984716 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.984655    2556 scope.go:117] "RemoveContainer" containerID="e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"
Jul 28 16:23:30.984931 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:23:30.984916    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a\": container with ID starting with e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a not found: ID does not exist" containerID="e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"
Jul 28 16:23:30.984971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.984938    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a"} err="failed to get container status \"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a\": rpc error: code = NotFound desc = could not find container \"e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a\": container with ID starting with e509a82edd577a692df2854f95e5689d718cf5d11cfc14243a406737dfb1048a not found: ID does not exist"
Jul 28 16:23:30.984971 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.984953    2556 scope.go:117] "RemoveContainer" containerID="209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"
Jul 28 16:23:30.992411 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.992376    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:23:30.993477 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.993453    2556 scope.go:117] "RemoveContainer" containerID="fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"
Jul 28 16:23:30.996888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:30.996862    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7c1b1-predictor-5f679bf9f-px27h"]
Jul 28 16:23:31.002586 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.002563    2556 scope.go:117] "RemoveContainer" containerID="209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"
Jul 28 16:23:31.002893 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:23:31.002875    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9\": container with ID starting with 209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9 not found: ID does not exist" containerID="209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"
Jul 28 16:23:31.002939 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.002903    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9"} err="failed to get container status \"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9\": rpc error: code = NotFound desc = could not find container \"209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9\": container with ID starting with 209eda3267c9e732188c0e4592227095d4ef5522a87f31612979546c41ebd8f9 not found: ID does not exist"
Jul 28 16:23:31.002939 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.002921    2556 scope.go:117] "RemoveContainer" containerID="fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"
Jul 28 16:23:31.003195 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:23:31.003176    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4\": container with ID starting with fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4 not found: ID does not exist" containerID="fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"
Jul 28 16:23:31.003237 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.003202    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4"} err="failed to get container status \"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4\": rpc error: code = NotFound desc = could not find container \"fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4\": container with ID starting with fb6fe05f6ad6d9c151f7525a68331e2247187bdf3f7f14b53ef6dccb31e401e4 not found: ID does not exist"
Jul 28 16:23:31.008752 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.008723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:23:31.013141 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.013114    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7c1b1-predictor-7f58bd6c46-mskvb"]
Jul 28 16:23:31.463579 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.463545    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" path="/var/lib/kubelet/pods/3534c90a-ca82-4e2c-8964-cd2439ba2fee/volumes"
Jul 28 16:23:31.464033 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:31.464016    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" path="/var/lib/kubelet/pods/f8e0ce44-1af6-49d1-80f6-2edca34b926a/volumes"
Jul 28 16:23:33.957172 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:33.957144    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:23:33.957691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:33.957664    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:23:34.962850 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:34.962822    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:23:34.963400 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:34.963273    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:23:43.958543 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:43.958501    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:23:44.964219 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:44.964178    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:23:53.958091 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:53.958048    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:23:54.963464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:23:54.963418    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:24:03.958135 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:24:03.958094    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:24:04.963288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:24:04.963240    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:24:13.959337 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:24:13.959303    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:24:14.964100 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:24:14.964072    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:26:15.429147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:26:15.429119    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:26:15.431310 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:26:15.431291    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:31:15.461554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:15.461523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:31:15.464334 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:15.464308    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:31:41.384605 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.384464    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:31:41.385121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385094    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kube-rbac-proxy"
Jul 28 16:31:41.385121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385116    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kube-rbac-proxy"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385139    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385149    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385179    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kube-rbac-proxy"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385189    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kube-rbac-proxy"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385208    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container"
Jul 28 16:31:41.385246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385217    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container"
Jul 28 16:31:41.385558 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385326    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kserve-container"
Jul 28 16:31:41.385558 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385335    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kube-rbac-proxy"
Jul 28 16:31:41.385558 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385342    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0ce44-1af6-49d1-80f6-2edca34b926a" containerName="kserve-container"
Jul 28 16:31:41.385558 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.385349    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3534c90a-ca82-4e2c-8964-cd2439ba2fee" containerName="kube-rbac-proxy"
Jul 28 16:31:41.396554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.395265    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.398275 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.398214    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a9ea4-predictor-serving-cert\""
Jul 28 16:31:41.398670 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.398644    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a9ea4-kube-rbac-proxy-sar-config\""
Jul 28 16:31:41.403688 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.403662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:31:41.409910 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.409884    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:31:41.410131 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.410094    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" containerID="cri-o://f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4" gracePeriod=30
Jul 28 16:31:41.410212 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.410173    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kube-rbac-proxy" containerID="cri-o://c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1" gracePeriod=30
Jul 28 16:31:41.481117 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.481079    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:31:41.484760 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.484735    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.487238 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.487211    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9ea4-predictor-serving-cert\""
Jul 28 16:31:41.487357 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.487241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9ea4-kube-rbac-proxy-sar-config\""
Jul 28 16:31:41.496042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.496014    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:31:41.532162 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.532134    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:31:41.532371 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.532349    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" containerID="cri-o://a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1" gracePeriod=30
Jul 28 16:31:41.532442 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.532391    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kube-rbac-proxy" containerID="cri-o://8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2" gracePeriod=30
Jul 28 16:31:41.543863 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.543826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.544015 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.543886    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2l92\" (UniqueName: \"kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.544083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.544035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.644853 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.644771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.644853 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.644839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.645059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.644927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.645059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.644956    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.645059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.645003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2l92\" (UniqueName: \"kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.645219 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.645061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwb5\" (UniqueName: \"kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.645674 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.645646    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.647754 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.647729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.653059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.653037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2l92\" (UniqueName: \"kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92\") pod \"success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") " pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.709391 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.709341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:41.746059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.746003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwb5\" (UniqueName: \"kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.746254 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.746071    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.746254 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.746172    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.747115 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.747083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.749037 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.749005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.754933 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.754903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwb5\" (UniqueName: \"kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5\") pod \"error-404-isvc-a9ea4-predictor-756b95979d-vf9xt\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") " pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.796902 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.796866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:41.855587 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.855495    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:31:41.857928 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:31:41.857855    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abc91bb_7c02_4a45_9f77_54df554fe14e.slice/crio-aeff8d113b47f68d6e693c6cdad27f817a91ffbe3f9df9168fbe5bac09aa14d7 WatchSource:0}: Error finding container aeff8d113b47f68d6e693c6cdad27f817a91ffbe3f9df9168fbe5bac09aa14d7: Status 404 returned error can't find the container with id aeff8d113b47f68d6e693c6cdad27f817a91ffbe3f9df9168fbe5bac09aa14d7
Jul 28 16:31:41.950502 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:41.950478    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:31:41.959910 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:31:41.959876    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6b0fb6_302a_4184_b950_ec0c0ed7f4e0.slice/crio-75d2ae12e6bf9b1cf618e6f9133a72bb0ee838468316f36d8517f87360f43ff4 WatchSource:0}: Error finding container 75d2ae12e6bf9b1cf618e6f9133a72bb0ee838468316f36d8517f87360f43ff4: Status 404 returned error can't find the container with id 75d2ae12e6bf9b1cf618e6f9133a72bb0ee838468316f36d8517f87360f43ff4
Jul 28 16:31:42.409451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.409417    2556 generic.go:356] "Generic (PLEG): container finished" podID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerID="c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1" exitCode=2
Jul 28 16:31:42.409893 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.409491    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerDied","Data":"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"}
Jul 28 16:31:42.411377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.411357    2556 generic.go:356] "Generic (PLEG): container finished" podID="9cec1195-36c6-4155-838e-f16d928c184b" containerID="8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2" exitCode=2
Jul 28 16:31:42.411469 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.411422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerDied","Data":"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"}
Jul 28 16:31:42.413361 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.413334    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerStarted","Data":"725f221b0543ab9023a279ccb200c9807d862df6a759dde4aadcbff8fbac3709"}
Jul 28 16:31:42.413473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.413387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerStarted","Data":"da1add7b8ce04de4b67dd7d211b2ca62c82ae5fea187a0e393db9c52e136738b"}
Jul 28 16:31:42.413473 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.413420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerStarted","Data":"75d2ae12e6bf9b1cf618e6f9133a72bb0ee838468316f36d8517f87360f43ff4"}
Jul 28 16:31:42.413589 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.413533    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:42.415083 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.415065    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerStarted","Data":"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"}
Jul 28 16:31:42.415181 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.415105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerStarted","Data":"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"}
Jul 28 16:31:42.415181 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.415136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerStarted","Data":"aeff8d113b47f68d6e693c6cdad27f817a91ffbe3f9df9168fbe5bac09aa14d7"}
Jul 28 16:31:42.415181 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.415161    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:42.434062 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.434010    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podStartSLOduration=1.4339953269999999 podStartE2EDuration="1.433995327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:42.432868056 +0000 UTC m=+1827.483672128" watchObservedRunningTime="2026-07-28 16:31:42.433995327 +0000 UTC m=+1827.484799400"
Jul 28 16:31:42.453185 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:42.453134    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podStartSLOduration=1.453116455 podStartE2EDuration="1.453116455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:31:42.45206096 +0000 UTC m=+1827.502865034" watchObservedRunningTime="2026-07-28 16:31:42.453116455 +0000 UTC m=+1827.503920528"
Jul 28 16:31:43.421100 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:43.421066    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:43.421572 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:43.421133    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:43.422123 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:43.422092    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:31:43.422123 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:43.422106    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:31:44.425420 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:44.425379    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:31:44.425913 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:44.425437    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:31:44.582610 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:44.582565    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 28 16:31:44.586986 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:44.586954    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 16:31:44.958477 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:44.958450    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:31:45.079700 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.079558    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") pod \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") "
Jul 28 16:31:45.079700 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.079607    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config\") pod \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") "
Jul 28 16:31:45.079945 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.079731    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhjz8\" (UniqueName: \"kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8\") pod \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\" (UID: \"af2e7a73-8319-4118-a799-4d8a2fdfd4e7\") "
Jul 28 16:31:45.080050 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.080020    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af2e7a73-8319-4118-a799-4d8a2fdfd4e7" (UID: "af2e7a73-8319-4118-a799-4d8a2fdfd4e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:45.081858 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.081832    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af2e7a73-8319-4118-a799-4d8a2fdfd4e7" (UID: "af2e7a73-8319-4118-a799-4d8a2fdfd4e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:45.081970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.081879    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8" (OuterVolumeSpecName: "kube-api-access-lhjz8") pod "af2e7a73-8319-4118-a799-4d8a2fdfd4e7" (UID: "af2e7a73-8319-4118-a799-4d8a2fdfd4e7"). InnerVolumeSpecName "kube-api-access-lhjz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:45.181124 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.181075    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjz8\" (UniqueName: \"kubernetes.io/projected/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-api-access-lhjz8\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:45.181124 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.181114    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:45.181124 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.181125    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af2e7a73-8319-4118-a799-4d8a2fdfd4e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:45.432111 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.432077    2556 generic.go:356] "Generic (PLEG): container finished" podID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerID="f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4" exitCode=0
Jul 28 16:31:45.432547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.432129    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"
Jul 28 16:31:45.432547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.432153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerDied","Data":"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"}
Jul 28 16:31:45.432547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.432228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l" event={"ID":"af2e7a73-8319-4118-a799-4d8a2fdfd4e7","Type":"ContainerDied","Data":"ac3a67082dbf47eecde3837bf5e72ca51481f78505ce73d5e2097b9deb1511e0"}
Jul 28 16:31:45.432547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.432253    2556 scope.go:117] "RemoveContainer" containerID="c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"
Jul 28 16:31:45.442137 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.442114    2556 scope.go:117] "RemoveContainer" containerID="f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"
Jul 28 16:31:45.450844 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.450819    2556 scope.go:117] "RemoveContainer" containerID="c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"
Jul 28 16:31:45.451215 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:31:45.451194    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1\": container with ID starting with c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1 not found: ID does not exist" containerID="c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"
Jul 28 16:31:45.451296 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.451227    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1"} err="failed to get container status \"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1\": rpc error: code = NotFound desc = could not find container \"c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1\": container with ID starting with c89e159eb003d4e29ae12f6fd92515e775ce37f569d8a5ef37ada82b3b2fc1c1 not found: ID does not exist"
Jul 28 16:31:45.451296 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.451247    2556 scope.go:117] "RemoveContainer" containerID="f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"
Jul 28 16:31:45.451573 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:31:45.451552    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4\": container with ID starting with f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4 not found: ID does not exist" containerID="f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"
Jul 28 16:31:45.451711 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.451578    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4"} err="failed to get container status \"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4\": rpc error: code = NotFound desc = could not find container \"f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4\": container with ID starting with f4ab71d57a125592760b8cbe2f36cbd240e426a4365c674d258c41e81de4b0d4 not found: ID does not exist"
Jul 28 16:31:45.455060 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.455033    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:31:45.463671 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.463613    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e5aa9-predictor-6d94bd5dd5-7g25l"]
Jul 28 16:31:45.591979 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.591938    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 28 16:31:45.597016 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.596980    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 28 16:31:45.898260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:45.898234    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:31:46.089749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.089659    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc59p\" (UniqueName: \"kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p\") pod \"9cec1195-36c6-4155-838e-f16d928c184b\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") "
Jul 28 16:31:46.089749 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.089713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") pod \"9cec1195-36c6-4155-838e-f16d928c184b\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") "
Jul 28 16:31:46.089953 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.089811    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config\") pod \"9cec1195-36c6-4155-838e-f16d928c184b\" (UID: \"9cec1195-36c6-4155-838e-f16d928c184b\") "
Jul 28 16:31:46.090169 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.090142    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9cec1195-36c6-4155-838e-f16d928c184b" (UID: "9cec1195-36c6-4155-838e-f16d928c184b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:31:46.091855 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.091829    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p" (OuterVolumeSpecName: "kube-api-access-bc59p") pod "9cec1195-36c6-4155-838e-f16d928c184b" (UID: "9cec1195-36c6-4155-838e-f16d928c184b"). InnerVolumeSpecName "kube-api-access-bc59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:31:46.091958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.091858    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9cec1195-36c6-4155-838e-f16d928c184b" (UID: "9cec1195-36c6-4155-838e-f16d928c184b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:31:46.190690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.190648    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc59p\" (UniqueName: \"kubernetes.io/projected/9cec1195-36c6-4155-838e-f16d928c184b-kube-api-access-bc59p\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:46.190690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.190681    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cec1195-36c6-4155-838e-f16d928c184b-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:46.190690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.190693    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cec1195-36c6-4155-838e-f16d928c184b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:31:46.439525 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.439491    2556 generic.go:356] "Generic (PLEG): container finished" podID="9cec1195-36c6-4155-838e-f16d928c184b" containerID="a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1" exitCode=0
Jul 28 16:31:46.440057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.439545    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"
Jul 28 16:31:46.440057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.439583    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerDied","Data":"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"}
Jul 28 16:31:46.440057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.439674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6" event={"ID":"9cec1195-36c6-4155-838e-f16d928c184b","Type":"ContainerDied","Data":"5120de93a0a5401d75007218f2ed882553d5aae91db0f3258a1d03dd4f7ca133"}
Jul 28 16:31:46.440057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.439721    2556 scope.go:117] "RemoveContainer" containerID="8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"
Jul 28 16:31:46.449936 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.449905    2556 scope.go:117] "RemoveContainer" containerID="a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"
Jul 28 16:31:46.458456 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.458436    2556 scope.go:117] "RemoveContainer" containerID="8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"
Jul 28 16:31:46.458777 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:31:46.458754    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2\": container with ID starting with 8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2 not found: ID does not exist" containerID="8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"
Jul 28 16:31:46.458861 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.458784    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2"} err="failed to get container status \"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2\": rpc error: code = NotFound desc = could not find container \"8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2\": container with ID starting with 8259204fd148a3d51b3cc7522d936bb9f0b0398208ccf432f3816fe5690236d2 not found: ID does not exist"
Jul 28 16:31:46.458861 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.458802    2556 scope.go:117] "RemoveContainer" containerID="a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"
Jul 28 16:31:46.459041 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:31:46.459024    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1\": container with ID starting with a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1 not found: ID does not exist" containerID="a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"
Jul 28 16:31:46.459078 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.459049    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1"} err="failed to get container status \"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1\": rpc error: code = NotFound desc = could not find container \"a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1\": container with ID starting with a67035df2ae7aee1c0889e8d46e5689bb341da16eb3ee8b9c065d18ea7e1bbe1 not found: ID does not exist"
Jul 28 16:31:46.465350 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.465321    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:31:46.469941 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:46.469914    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e5aa9-predictor-5f4f89b-pj4n6"]
Jul 28 16:31:47.463365 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:47.463329    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cec1195-36c6-4155-838e-f16d928c184b" path="/var/lib/kubelet/pods/9cec1195-36c6-4155-838e-f16d928c184b/volumes"
Jul 28 16:31:47.463798 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:47.463782    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" path="/var/lib/kubelet/pods/af2e7a73-8319-4118-a799-4d8a2fdfd4e7/volumes"
Jul 28 16:31:49.431703 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:49.431675    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:31:49.432124 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:49.432033    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:31:49.432194 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:49.432162    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:31:49.432539 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:49.432510    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:31:59.433175 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:59.433136    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:31:59.433570 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:31:59.433136    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:32:09.432683 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:09.432644    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:32:09.433093 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:09.432641    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:32:19.432931 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:19.432888    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 16:32:19.433372 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:19.432887    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 16:32:29.432995 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:29.432962    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:32:29.435379 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:29.433546    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:32:41.317482 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.317387    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:32:41.317919 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.317640    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" containerID="cri-o://3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136" gracePeriod=30
Jul 28 16:32:41.317919 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.317694    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kube-rbac-proxy" containerID="cri-o://71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53" gracePeriod=30
Jul 28 16:32:41.385048 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.385005    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:32:41.385423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.385367    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" containerID="cri-o://21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b" gracePeriod=30
Jul 28 16:32:41.385675 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.385449    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kube-rbac-proxy" containerID="cri-o://b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53" gracePeriod=30
Jul 28 16:32:41.408492 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408465    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:32:41.408922 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408908    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container"
Jul 28 16:32:41.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408926    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container"
Jul 28 16:32:41.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408949    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kube-rbac-proxy"
Jul 28 16:32:41.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408954    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kube-rbac-proxy"
Jul 28 16:32:41.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408966    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container"
Jul 28 16:32:41.408970 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408971    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408982    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kube-rbac-proxy"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.408987    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kube-rbac-proxy"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.409048    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kube-rbac-proxy"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.409056    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kserve-container"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.409064    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2e7a73-8319-4118-a799-4d8a2fdfd4e7" containerName="kube-rbac-proxy"
Jul 28 16:32:41.409125 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.409069    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cec1195-36c6-4155-838e-f16d928c184b" containerName="kserve-container"
Jul 28 16:32:41.412506 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.412485    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.415290 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.415038    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3a3d-predictor-serving-cert\""
Jul 28 16:32:41.415290 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.415051    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d3a3d-kube-rbac-proxy-sar-config\""
Jul 28 16:32:41.426827 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.426800    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:32:41.516585 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.516550    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.516859 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.516704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xqj\" (UniqueName: \"kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.516859 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.516795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.526288 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.526259    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:32:41.530070 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.530047    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.532602 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.532574    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d3a3d-predictor-serving-cert\""
Jul 28 16:32:41.532732 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.532701    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d3a3d-kube-rbac-proxy-sar-config\""
Jul 28 16:32:41.542318 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.542290    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:32:41.617833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.617798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.617833 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.617836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.618095 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.617882    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.618095 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.617904    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98d6\" (UniqueName: \"kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.618095 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.617931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.618095 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.618038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xqj\" (UniqueName: \"kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.618600 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.618578    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.620560 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.620536    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.626585 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.626555    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xqj\" (UniqueName: \"kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj\") pod \"success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") " pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.718720 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.718685    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.718922 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.718735    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98d6\" (UniqueName: \"kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.718922 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.718823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.719028 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:41.718994    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-serving-cert: secret "error-404-isvc-d3a3d-predictor-serving-cert" not found
Jul 28 16:32:41.719064 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:41.719059    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls podName:0756a0df-0845-4c28-ab0c-e213f648fa1a nodeName:}" failed. No retries permitted until 2026-07-28 16:32:42.219039515 +0000 UTC m=+1887.269843577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls") pod "error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" (UID: "0756a0df-0845-4c28-ab0c-e213f648fa1a") : secret "error-404-isvc-d3a3d-predictor-serving-cert" not found
Jul 28 16:32:41.719428 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.719406    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.723560 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.723531    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerID="b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53" exitCode=2
Jul 28 16:32:41.723774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.723578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerDied","Data":"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"}
Jul 28 16:32:41.723774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.723540    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:41.725440 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.725411    2556 generic.go:356] "Generic (PLEG): container finished" podID="3369701d-a56c-44c5-bff1-8ecd97555590" containerID="71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53" exitCode=2
Jul 28 16:32:41.725554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.725446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerDied","Data":"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"}
Jul 28 16:32:41.727555 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.727532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98d6\" (UniqueName: \"kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:41.867701 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:41.867676    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:32:41.869466 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:32:41.869401    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d71f4da_75b7_4681_8b59_daeff6441f33.slice/crio-ccb76fab9f75d93328c2e8d2b1bd30d32fc3ccb4e80469faa6c449fb3e8c1575 WatchSource:0}: Error finding container ccb76fab9f75d93328c2e8d2b1bd30d32fc3ccb4e80469faa6c449fb3e8c1575: Status 404 returned error can't find the container with id ccb76fab9f75d93328c2e8d2b1bd30d32fc3ccb4e80469faa6c449fb3e8c1575
Jul 28 16:32:42.224896 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.224803    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:42.227361 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.227343    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") pod \"error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") " pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:42.441691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.441610    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:42.585864 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.585838    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:32:42.587720 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:32:42.587691    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0756a0df_0845_4c28_ab0c_e213f648fa1a.slice/crio-00363aa2a201c12e3f8db6f34af58d0ba209a09861c91893bbba489f85a0ebc5 WatchSource:0}: Error finding container 00363aa2a201c12e3f8db6f34af58d0ba209a09861c91893bbba489f85a0ebc5: Status 404 returned error can't find the container with id 00363aa2a201c12e3f8db6f34af58d0ba209a09861c91893bbba489f85a0ebc5
Jul 28 16:32:42.732341 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.732297    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerStarted","Data":"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"}
Jul 28 16:32:42.732530 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.732377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerStarted","Data":"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"}
Jul 28 16:32:42.732530 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.732415    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:42.732530 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.732451    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerStarted","Data":"00363aa2a201c12e3f8db6f34af58d0ba209a09861c91893bbba489f85a0ebc5"}
Jul 28 16:32:42.734100 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.734077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerStarted","Data":"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"}
Jul 28 16:32:42.734238 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.734130    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerStarted","Data":"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"}
Jul 28 16:32:42.734238 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.734172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerStarted","Data":"ccb76fab9f75d93328c2e8d2b1bd30d32fc3ccb4e80469faa6c449fb3e8c1575"}
Jul 28 16:32:42.734238 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.734205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:42.734238 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.734232    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:42.735661 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.735637    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:32:42.752605 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.752475    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podStartSLOduration=1.752450977 podStartE2EDuration="1.752450977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:42.750529548 +0000 UTC m=+1887.801333617" watchObservedRunningTime="2026-07-28 16:32:42.752450977 +0000 UTC m=+1887.803255050"
Jul 28 16:32:42.776583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:42.772887    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podStartSLOduration=1.772865031 podStartE2EDuration="1.772865031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:42.771282417 +0000 UTC m=+1887.822086482" watchObservedRunningTime="2026-07-28 16:32:42.772865031 +0000 UTC m=+1887.823669105"
Jul 28 16:32:43.739045 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:43.738998    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:43.739045 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:43.739008    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:32:43.740378 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:43.740348    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:32:43.952638 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:43.952584    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 28 16:32:43.957966 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:43.957937    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 16:32:44.745313 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:44.745268    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:32:44.958943 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:44.958900    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 28 16:32:44.963313 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:44.963282    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 16:32:45.252575 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.252532    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:32:45.352185 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.352150    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config\") pod \"ec669095-1542-45fb-bfc9-cf8ca7b37332\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") "
Jul 28 16:32:45.352382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.352198    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnrfd\" (UniqueName: \"kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd\") pod \"ec669095-1542-45fb-bfc9-cf8ca7b37332\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") "
Jul 28 16:32:45.352382 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.352274    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls\") pod \"ec669095-1542-45fb-bfc9-cf8ca7b37332\" (UID: \"ec669095-1542-45fb-bfc9-cf8ca7b37332\") "
Jul 28 16:32:45.352634 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.352593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec669095-1542-45fb-bfc9-cf8ca7b37332" (UID: "ec669095-1542-45fb-bfc9-cf8ca7b37332"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:32:45.354579 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.354547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd" (OuterVolumeSpecName: "kube-api-access-vnrfd") pod "ec669095-1542-45fb-bfc9-cf8ca7b37332" (UID: "ec669095-1542-45fb-bfc9-cf8ca7b37332"). InnerVolumeSpecName "kube-api-access-vnrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:32:45.354579 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.354565    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec669095-1542-45fb-bfc9-cf8ca7b37332" (UID: "ec669095-1542-45fb-bfc9-cf8ca7b37332"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:32:45.370882 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.370854    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:32:45.453404 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453369    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4bg9\" (UniqueName: \"kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9\") pod \"3369701d-a56c-44c5-bff1-8ecd97555590\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") "
Jul 28 16:32:45.453566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453444    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls\") pod \"3369701d-a56c-44c5-bff1-8ecd97555590\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") "
Jul 28 16:32:45.453566 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453533    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config\") pod \"3369701d-a56c-44c5-bff1-8ecd97555590\" (UID: \"3369701d-a56c-44c5-bff1-8ecd97555590\") "
Jul 28 16:32:45.453840 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453821    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.453908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453846    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrfd\" (UniqueName: \"kubernetes.io/projected/ec669095-1542-45fb-bfc9-cf8ca7b37332-kube-api-access-vnrfd\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.453908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453859    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec669095-1542-45fb-bfc9-cf8ca7b37332-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.453908 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.453863    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3369701d-a56c-44c5-bff1-8ecd97555590" (UID: "3369701d-a56c-44c5-bff1-8ecd97555590"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:32:45.455752 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.455726    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9" (OuterVolumeSpecName: "kube-api-access-l4bg9") pod "3369701d-a56c-44c5-bff1-8ecd97555590" (UID: "3369701d-a56c-44c5-bff1-8ecd97555590"). InnerVolumeSpecName "kube-api-access-l4bg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:32:45.455868 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.455780    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3369701d-a56c-44c5-bff1-8ecd97555590" (UID: "3369701d-a56c-44c5-bff1-8ecd97555590"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:32:45.555025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.554793    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3369701d-a56c-44c5-bff1-8ecd97555590-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.555025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.554830    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4bg9\" (UniqueName: \"kubernetes.io/projected/3369701d-a56c-44c5-bff1-8ecd97555590-kube-api-access-l4bg9\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.555025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.554845    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3369701d-a56c-44c5-bff1-8ecd97555590-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:45.752796 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.752716    2556 generic.go:356] "Generic (PLEG): container finished" podID="3369701d-a56c-44c5-bff1-8ecd97555590" containerID="3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136" exitCode=0
Jul 28 16:32:45.752796 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.752777    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"
Jul 28 16:32:45.752796 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.752768    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerDied","Data":"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"}
Jul 28 16:32:45.753320 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.752837    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm" event={"ID":"3369701d-a56c-44c5-bff1-8ecd97555590","Type":"ContainerDied","Data":"bfc6fa1cc5432139debec51bd0bab678ba328ff65e9c15b973cf91284b994d2f"}
Jul 28 16:32:45.753320 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.752866    2556 scope.go:117] "RemoveContainer" containerID="71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"
Jul 28 16:32:45.754761 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.754739    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerID="21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b" exitCode=0
Jul 28 16:32:45.754883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.754773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerDied","Data":"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"}
Jul 28 16:32:45.754883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.754785    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"
Jul 28 16:32:45.754883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.754831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk" event={"ID":"ec669095-1542-45fb-bfc9-cf8ca7b37332","Type":"ContainerDied","Data":"dbdfa250d0b545e287170c9e7683681edbeb33a0bd7e22cfd73aef35c5619955"}
Jul 28 16:32:45.762304 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.762280    2556 scope.go:117] "RemoveContainer" containerID="3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"
Jul 28 16:32:45.770431 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.770407    2556 scope.go:117] "RemoveContainer" containerID="71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"
Jul 28 16:32:45.770744 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:45.770725    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53\": container with ID starting with 71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53 not found: ID does not exist" containerID="71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"
Jul 28 16:32:45.770786 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.770753    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53"} err="failed to get container status \"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53\": rpc error: code = NotFound desc = could not find container \"71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53\": container with ID starting with 71d08690aa56e4795681f3ea28ea0256b8fccfe6a2f95e9391c9cb320d321d53 not found: ID does not exist"
Jul 28 16:32:45.770786 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.770770    2556 scope.go:117] "RemoveContainer" containerID="3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"
Jul 28 16:32:45.771023 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:45.771004    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136\": container with ID starting with 3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136 not found: ID does not exist" containerID="3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"
Jul 28 16:32:45.771059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.771031    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136"} err="failed to get container status \"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136\": rpc error: code = NotFound desc = could not find container \"3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136\": container with ID starting with 3ca8ba74a48f24c8c89cb6ceffcf1c767f2388b7fff6df2c2c8eda03466dc136 not found: ID does not exist"
Jul 28 16:32:45.771059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.771054    2556 scope.go:117] "RemoveContainer" containerID="b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"
Jul 28 16:32:45.774217 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.774191    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:32:45.777531 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.777505    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c3dc2-predictor-58b487689c-z5qjm"]
Jul 28 16:32:45.782779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.782745    2556 scope.go:117] "RemoveContainer" containerID="21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"
Jul 28 16:32:45.788140 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.788111    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:32:45.791429 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.791401    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3dc2-predictor-6987f9bc88-c96rk"]
Jul 28 16:32:45.793047 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.793030    2556 scope.go:117] "RemoveContainer" containerID="b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"
Jul 28 16:32:45.793340 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:45.793322    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53\": container with ID starting with b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53 not found: ID does not exist" containerID="b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"
Jul 28 16:32:45.793407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.793349    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53"} err="failed to get container status \"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53\": rpc error: code = NotFound desc = could not find container \"b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53\": container with ID starting with b7947ea9cbe88143a11c8212db202816dc5156e8c9cc72b9a9fba7be30227d53 not found: ID does not exist"
Jul 28 16:32:45.793407 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.793371    2556 scope.go:117] "RemoveContainer" containerID="21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"
Jul 28 16:32:45.793650 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:45.793610    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b\": container with ID starting with 21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b not found: ID does not exist" containerID="21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"
Jul 28 16:32:45.793697 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:45.793660    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b"} err="failed to get container status \"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b\": rpc error: code = NotFound desc = could not find container \"21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b\": container with ID starting with 21a020ae877f4deb3d3ef35b337edeee3e0927c2041ec310a42cf971b11dab9b not found: ID does not exist"
Jul 28 16:32:47.463464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:47.463426    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" path="/var/lib/kubelet/pods/3369701d-a56c-44c5-bff1-8ecd97555590/volumes"
Jul 28 16:32:47.464046 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:47.464029    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" path="/var/lib/kubelet/pods/ec669095-1542-45fb-bfc9-cf8ca7b37332/volumes"
Jul 28 16:32:48.743543 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:48.743506    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:32:48.744245 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:48.744179    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:32:49.751011 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:49.750973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:32:49.751661 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:49.751602    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:32:51.588109 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.588074    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:32:51.588548 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.588292    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container" containerID="cri-o://3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7" gracePeriod=30
Jul 28 16:32:51.588548 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.588338    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kube-rbac-proxy" containerID="cri-o://9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da" gracePeriod=30
Jul 28 16:32:51.622664 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.622610    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:32:51.623035 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623022    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kube-rbac-proxy"
Jul 28 16:32:51.623035 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623036    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kube-rbac-proxy"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623051    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623057    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623071    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kube-rbac-proxy"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623077    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kube-rbac-proxy"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623085    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container"
Jul 28 16:32:51.623121 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623090    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container"
Jul 28 16:32:51.623302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623143    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kserve-container"
Jul 28 16:32:51.623302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623151    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kube-rbac-proxy"
Jul 28 16:32:51.623302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623161    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec669095-1542-45fb-bfc9-cf8ca7b37332" containerName="kserve-container"
Jul 28 16:32:51.623302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.623167    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3369701d-a56c-44c5-bff1-8ecd97555590" containerName="kube-rbac-proxy"
Jul 28 16:32:51.627919 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.627897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.630728 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.630707    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49c25-kube-rbac-proxy-sar-config\""
Jul 28 16:32:51.630842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.630731    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49c25-predictor-serving-cert\""
Jul 28 16:32:51.638289 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.638264    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:32:51.703051 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.703022    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:32:51.703380 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.703353    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container" containerID="cri-o://da1add7b8ce04de4b67dd7d211b2ca62c82ae5fea187a0e393db9c52e136738b" gracePeriod=30
Jul 28 16:32:51.703465 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.703431    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kube-rbac-proxy" containerID="cri-o://725f221b0543ab9023a279ccb200c9807d862df6a759dde4aadcbff8fbac3709" gracePeriod=30
Jul 28 16:32:51.709277 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.709242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.709400 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.709346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5nk7\" (UniqueName: \"kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.709464 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.709442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.735239 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.735201    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:32:51.739244 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.739218    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.742038 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.742006    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49c25-kube-rbac-proxy-sar-config\""
Jul 28 16:32:51.742199 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.742006    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49c25-predictor-serving-cert\""
Jul 28 16:32:51.752543 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.752510    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:32:51.790434 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.790403    2556 generic.go:356] "Generic (PLEG): container finished" podID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerID="9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da" exitCode=2
Jul 28 16:32:51.790605 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.790458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerDied","Data":"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"}
Jul 28 16:32:51.810920 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.810887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.811122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.810944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.811122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.810986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.811122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.811049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.811122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.811073    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwtj\" (UniqueName: \"kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.811122 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.811104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5nk7\" (UniqueName: \"kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.811561 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.811538    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.813557 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.813533    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.822750 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.822697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5nk7\" (UniqueName: \"kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7\") pod \"success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") " pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:51.912557 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.912520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.912557 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.912562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwtj\" (UniqueName: \"kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.912868 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.912652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.913354 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.913330    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.915282 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.915257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.921458 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.921434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwtj\" (UniqueName: \"kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj\") pod \"error-404-isvc-49c25-predictor-5c44655c88-2xw8x\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") " pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:51.939838 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:51.939803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:52.055136 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.055079    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:52.085842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.085811    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:32:52.088497 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:32:52.088470    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddebf89_052d_4d13_9c5c_0fef273bb273.slice/crio-0ff6a0577865eaff968d38ffca95f9bced95a8c79af02663c28e34494120b295 WatchSource:0}: Error finding container 0ff6a0577865eaff968d38ffca95f9bced95a8c79af02663c28e34494120b295: Status 404 returned error can't find the container with id 0ff6a0577865eaff968d38ffca95f9bced95a8c79af02663c28e34494120b295
Jul 28 16:32:52.207716 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.207684    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:32:52.209587 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:32:52.209561    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93032262_4d31_45cb_9ad2_161cab3accbd.slice/crio-2d2ff8686099e1a6d07d9213d87bf6d59ad06f27f940d179135b6ce1c6034d88 WatchSource:0}: Error finding container 2d2ff8686099e1a6d07d9213d87bf6d59ad06f27f940d179135b6ce1c6034d88: Status 404 returned error can't find the container with id 2d2ff8686099e1a6d07d9213d87bf6d59ad06f27f940d179135b6ce1c6034d88
Jul 28 16:32:52.797809 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.797771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerStarted","Data":"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"}
Jul 28 16:32:52.798315 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.797873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerStarted","Data":"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"}
Jul 28 16:32:52.798315 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.797911    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:52.798315 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.797939    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerStarted","Data":"2d2ff8686099e1a6d07d9213d87bf6d59ad06f27f940d179135b6ce1c6034d88"}
Jul 28 16:32:52.799638 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.799592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerStarted","Data":"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"}
Jul 28 16:32:52.799779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.799669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerStarted","Data":"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"}
Jul 28 16:32:52.799779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.799700    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerStarted","Data":"0ff6a0577865eaff968d38ffca95f9bced95a8c79af02663c28e34494120b295"}
Jul 28 16:32:52.799779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.799728    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:52.799779 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.799762    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:52.801515 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.801483    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:32:52.801614 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.801571    2556 generic.go:356] "Generic (PLEG): container finished" podID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerID="725f221b0543ab9023a279ccb200c9807d862df6a759dde4aadcbff8fbac3709" exitCode=2
Jul 28 16:32:52.801690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.801611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerDied","Data":"725f221b0543ab9023a279ccb200c9807d862df6a759dde4aadcbff8fbac3709"}
Jul 28 16:32:52.817582 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.817404    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podStartSLOduration=1.817385043 podStartE2EDuration="1.817385043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:52.816915483 +0000 UTC m=+1897.867719583" watchObservedRunningTime="2026-07-28 16:32:52.817385043 +0000 UTC m=+1897.868189118"
Jul 28 16:32:52.835901 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:52.835825    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podStartSLOduration=1.835802069 podStartE2EDuration="1.835802069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:32:52.835405756 +0000 UTC m=+1897.886209834" watchObservedRunningTime="2026-07-28 16:32:52.835802069 +0000 UTC m=+1897.886606144"
Jul 28 16:32:53.807233 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:53.807178    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:32:53.807691 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:53.807345    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:53.808799 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:53.808769    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:32:54.425966 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:54.425929    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 16:32:54.425966 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:54.425928    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 28 16:32:54.812168 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:54.812078    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:32:55.556074 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.556047    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:32:55.649342 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.649284    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls\") pod \"7abc91bb-7c02-4a45-9f77-54df554fe14e\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") "
Jul 28 16:32:55.649549 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.649357    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config\") pod \"7abc91bb-7c02-4a45-9f77-54df554fe14e\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") "
Jul 28 16:32:55.649549 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.649469    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2l92\" (UniqueName: \"kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92\") pod \"7abc91bb-7c02-4a45-9f77-54df554fe14e\" (UID: \"7abc91bb-7c02-4a45-9f77-54df554fe14e\") "
Jul 28 16:32:55.649883 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.649851    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7abc91bb-7c02-4a45-9f77-54df554fe14e" (UID: "7abc91bb-7c02-4a45-9f77-54df554fe14e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:32:55.652003 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.651956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92" (OuterVolumeSpecName: "kube-api-access-k2l92") pod "7abc91bb-7c02-4a45-9f77-54df554fe14e" (UID: "7abc91bb-7c02-4a45-9f77-54df554fe14e"). InnerVolumeSpecName "kube-api-access-k2l92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:32:55.652300 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.652270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7abc91bb-7c02-4a45-9f77-54df554fe14e" (UID: "7abc91bb-7c02-4a45-9f77-54df554fe14e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:32:55.750640 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.750572    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2l92\" (UniqueName: \"kubernetes.io/projected/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-api-access-k2l92\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:55.750835 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.750647    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abc91bb-7c02-4a45-9f77-54df554fe14e-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:55.750835 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.750665    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abc91bb-7c02-4a45-9f77-54df554fe14e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:55.819308 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.819254    2556 generic.go:356] "Generic (PLEG): container finished" podID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerID="da1add7b8ce04de4b67dd7d211b2ca62c82ae5fea187a0e393db9c52e136738b" exitCode=0
Jul 28 16:32:55.819761 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.819332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerDied","Data":"da1add7b8ce04de4b67dd7d211b2ca62c82ae5fea187a0e393db9c52e136738b"}
Jul 28 16:32:55.821250 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.821222    2556 generic.go:356] "Generic (PLEG): container finished" podID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerID="3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7" exitCode=0
Jul 28 16:32:55.821383 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.821284    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"
Jul 28 16:32:55.821383 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.821300    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerDied","Data":"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"}
Jul 28 16:32:55.821383 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.821359    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc" event={"ID":"7abc91bb-7c02-4a45-9f77-54df554fe14e","Type":"ContainerDied","Data":"aeff8d113b47f68d6e693c6cdad27f817a91ffbe3f9df9168fbe5bac09aa14d7"}
Jul 28 16:32:55.821495 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.821398    2556 scope.go:117] "RemoveContainer" containerID="9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"
Jul 28 16:32:55.831368 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.831344    2556 scope.go:117] "RemoveContainer" containerID="3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"
Jul 28 16:32:55.839895 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.839871    2556 scope.go:117] "RemoveContainer" containerID="9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"
Jul 28 16:32:55.840200 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:55.840175    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da\": container with ID starting with 9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da not found: ID does not exist" containerID="9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"
Jul 28 16:32:55.840273 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.840216    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da"} err="failed to get container status \"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da\": rpc error: code = NotFound desc = could not find container \"9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da\": container with ID starting with 9034c636eabcba133256e8f4ae48e81238c579dbd9131fe31e6bc2eda5d635da not found: ID does not exist"
Jul 28 16:32:55.840273 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.840241    2556 scope.go:117] "RemoveContainer" containerID="3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"
Jul 28 16:32:55.840491 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:32:55.840472    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7\": container with ID starting with 3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7 not found: ID does not exist" containerID="3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"
Jul 28 16:32:55.840535 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.840499    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7"} err="failed to get container status \"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7\": rpc error: code = NotFound desc = could not find container \"3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7\": container with ID starting with 3368a767c7b7fdfddb7d4a24ffb44de71551217e54e552336c995772dca31ec7 not found: ID does not exist"
Jul 28 16:32:55.845552 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.845510    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:32:55.846329 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.846303    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:32:55.850885 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.850858    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9ea4-predictor-64d69d9c55-2bfwc"]
Jul 28 16:32:55.952412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.952380    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config\") pod \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") "
Jul 28 16:32:55.952640 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.952428    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qwb5\" (UniqueName: \"kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5\") pod \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") "
Jul 28 16:32:55.952640 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.952472    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls\") pod \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\" (UID: \"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0\") "
Jul 28 16:32:55.952804 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.952776    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" (UID: "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:32:55.954750 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.954726    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5" (OuterVolumeSpecName: "kube-api-access-2qwb5") pod "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" (UID: "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0"). InnerVolumeSpecName "kube-api-access-2qwb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:32:55.954836 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:55.954755    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" (UID: "1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:32:56.053446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.053362    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:56.053446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.053392    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwb5\" (UniqueName: \"kubernetes.io/projected/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-kube-api-access-2qwb5\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:56.053446 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.053402    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:32:56.828080 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.828042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt" event={"ID":"1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0","Type":"ContainerDied","Data":"75d2ae12e6bf9b1cf618e6f9133a72bb0ee838468316f36d8517f87360f43ff4"}
Jul 28 16:32:56.828080 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.828061    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"
Jul 28 16:32:56.828652 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.828129    2556 scope.go:117] "RemoveContainer" containerID="725f221b0543ab9023a279ccb200c9807d862df6a759dde4aadcbff8fbac3709"
Jul 28 16:32:56.837583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.837560    2556 scope.go:117] "RemoveContainer" containerID="da1add7b8ce04de4b67dd7d211b2ca62c82ae5fea187a0e393db9c52e136738b"
Jul 28 16:32:56.851329 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.851300    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:32:56.853847 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:56.853798    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9ea4-predictor-756b95979d-vf9xt"]
Jul 28 16:32:57.462512 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:57.462474    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" path="/var/lib/kubelet/pods/1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0/volumes"
Jul 28 16:32:57.463072 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:57.463050    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" path="/var/lib/kubelet/pods/7abc91bb-7c02-4a45-9f77-54df554fe14e/volumes"
Jul 28 16:32:58.744696 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:58.744650    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:32:58.812096 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:58.812062    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:32:58.812528 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:58.812503    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:32:59.751686 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:59.751649    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:32:59.816683 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:59.816654    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:32:59.817059 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:32:59.817036    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:08.744939 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:08.744890    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:33:08.813546 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:08.813507    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:33:09.751941 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:09.751901    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:33:09.817982 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:09.817940    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:18.744846 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:18.744800    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 16:33:18.813555 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:18.813516    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:33:19.752186 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:19.752146    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:33:19.817555 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:19.817519    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:28.745799 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:28.745767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:33:28.812959 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:28.812924    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:33:29.752258 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:29.752220    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 16:33:29.817246 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:29.817203    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:33:38.812598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:38.812555    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:33:39.752760 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:39.752725    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:33:39.818326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:39.818297    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:33:48.814015 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:33:48.813983    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:36:15.493069 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:36:15.493042    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:36:15.496955 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:36:15.496934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:41:15.530724 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:41:15.530696    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:41:15.533711 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:41:15.533686    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:42:16.519367 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.519327    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:42:16.520035 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.519704    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" containerID="cri-o://b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58" gracePeriod=30
Jul 28 16:42:16.520035 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.519725    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kube-rbac-proxy" containerID="cri-o://58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59" gracePeriod=30
Jul 28 16:42:16.587690 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.587650    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:42:16.587946 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.587897    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" containerID="cri-o://39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06" gracePeriod=30
Jul 28 16:42:16.588026 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:16.587978    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kube-rbac-proxy" containerID="cri-o://735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489" gracePeriod=30
Jul 28 16:42:17.549598 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:17.549555    2556 generic.go:356] "Generic (PLEG): container finished" podID="93032262-4d31-45cb-9ad2-161cab3accbd" containerID="735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489" exitCode=2
Jul 28 16:42:17.550103 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:17.549655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerDied","Data":"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"}
Jul 28 16:42:17.551536 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:17.551504    2556 generic.go:356] "Generic (PLEG): container finished" podID="addebf89-052d-4d13-9c5c-0fef273bb273" containerID="58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59" exitCode=2
Jul 28 16:42:17.551666 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:17.551574    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerDied","Data":"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"}
Jul 28 16:42:18.807617 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:18.807572    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 28 16:42:18.812958 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:18.812920    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 16:42:19.812513 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:19.812474    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 16:42:19.817879 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:19.817847    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 28 16:42:19.996205 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:19.996134    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:42:19.999061 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:19.999038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:42:20.100988 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.100948    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rwtj\" (UniqueName: \"kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj\") pod \"93032262-4d31-45cb-9ad2-161cab3accbd\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") "
Jul 28 16:42:20.101180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101016    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config\") pod \"93032262-4d31-45cb-9ad2-161cab3accbd\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") "
Jul 28 16:42:20.101180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101045    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls\") pod \"93032262-4d31-45cb-9ad2-161cab3accbd\" (UID: \"93032262-4d31-45cb-9ad2-161cab3accbd\") "
Jul 28 16:42:20.101180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101066    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config\") pod \"addebf89-052d-4d13-9c5c-0fef273bb273\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") "
Jul 28 16:42:20.101180 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101091    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls\") pod \"addebf89-052d-4d13-9c5c-0fef273bb273\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") "
Jul 28 16:42:20.101377 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101275    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5nk7\" (UniqueName: \"kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7\") pod \"addebf89-052d-4d13-9c5c-0fef273bb273\" (UID: \"addebf89-052d-4d13-9c5c-0fef273bb273\") "
Jul 28 16:42:20.101475 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101449    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93032262-4d31-45cb-9ad2-161cab3accbd" (UID: "93032262-4d31-45cb-9ad2-161cab3accbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:20.101544 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101480    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "addebf89-052d-4d13-9c5c-0fef273bb273" (UID: "addebf89-052d-4d13-9c5c-0fef273bb273"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:42:20.101608 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101593    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93032262-4d31-45cb-9ad2-161cab3accbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.101685 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.101612    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/addebf89-052d-4d13-9c5c-0fef273bb273-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.103508 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.103483    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "addebf89-052d-4d13-9c5c-0fef273bb273" (UID: "addebf89-052d-4d13-9c5c-0fef273bb273"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:42:20.103508 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.103493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7" (OuterVolumeSpecName: "kube-api-access-d5nk7") pod "addebf89-052d-4d13-9c5c-0fef273bb273" (UID: "addebf89-052d-4d13-9c5c-0fef273bb273"). InnerVolumeSpecName "kube-api-access-d5nk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:42:20.103841 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.103816    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93032262-4d31-45cb-9ad2-161cab3accbd" (UID: "93032262-4d31-45cb-9ad2-161cab3accbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:42:20.103927 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.103820    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj" (OuterVolumeSpecName: "kube-api-access-4rwtj") pod "93032262-4d31-45cb-9ad2-161cab3accbd" (UID: "93032262-4d31-45cb-9ad2-161cab3accbd"). InnerVolumeSpecName "kube-api-access-4rwtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:42:20.202042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.202006    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/addebf89-052d-4d13-9c5c-0fef273bb273-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.202042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.202034    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5nk7\" (UniqueName: \"kubernetes.io/projected/addebf89-052d-4d13-9c5c-0fef273bb273-kube-api-access-d5nk7\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.202042 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.202046    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rwtj\" (UniqueName: \"kubernetes.io/projected/93032262-4d31-45cb-9ad2-161cab3accbd-kube-api-access-4rwtj\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.202260 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.202055    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93032262-4d31-45cb-9ad2-161cab3accbd-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:42:20.570614 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.570578    2556 generic.go:356] "Generic (PLEG): container finished" podID="93032262-4d31-45cb-9ad2-161cab3accbd" containerID="39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06" exitCode=0
Jul 28 16:42:20.570847 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.570673    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"
Jul 28 16:42:20.570847 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.570669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerDied","Data":"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"}
Jul 28 16:42:20.570847 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.570794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x" event={"ID":"93032262-4d31-45cb-9ad2-161cab3accbd","Type":"ContainerDied","Data":"2d2ff8686099e1a6d07d9213d87bf6d59ad06f27f940d179135b6ce1c6034d88"}
Jul 28 16:42:20.570847 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.570828    2556 scope.go:117] "RemoveContainer" containerID="735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"
Jul 28 16:42:20.580195 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.580158    2556 generic.go:356] "Generic (PLEG): container finished" podID="addebf89-052d-4d13-9c5c-0fef273bb273" containerID="b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58" exitCode=0
Jul 28 16:42:20.580339 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.580205    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerDied","Data":"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"}
Jul 28 16:42:20.580339 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.580293    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg" event={"ID":"addebf89-052d-4d13-9c5c-0fef273bb273","Type":"ContainerDied","Data":"0ff6a0577865eaff968d38ffca95f9bced95a8c79af02663c28e34494120b295"}
Jul 28 16:42:20.580339 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.580292    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"
Jul 28 16:42:20.589209 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.589189    2556 scope.go:117] "RemoveContainer" containerID="39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"
Jul 28 16:42:20.597952 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.597919    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:42:20.598456 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.598439    2556 scope.go:117] "RemoveContainer" containerID="735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"
Jul 28 16:42:20.598776 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:42:20.598753    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489\": container with ID starting with 735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489 not found: ID does not exist" containerID="735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"
Jul 28 16:42:20.598837 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.598785    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489"} err="failed to get container status \"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489\": rpc error: code = NotFound desc = could not find container \"735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489\": container with ID starting with 735fc97e23fac2152caadd7a8384d242928b675f7dfa070d5ef4d4779ad88489 not found: ID does not exist"
Jul 28 16:42:20.598837 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.598803    2556 scope.go:117] "RemoveContainer" containerID="39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"
Jul 28 16:42:20.599045 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:42:20.599022    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06\": container with ID starting with 39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06 not found: ID does not exist" containerID="39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"
Jul 28 16:42:20.599115 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.599055    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06"} err="failed to get container status \"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06\": rpc error: code = NotFound desc = could not find container \"39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06\": container with ID starting with 39a362533fa874d79455415063596f39446a2dae2e4b5cfcb0a4676a226d3e06 not found: ID does not exist"
Jul 28 16:42:20.599115 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.599078    2556 scope.go:117] "RemoveContainer" containerID="58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"
Jul 28 16:42:20.606203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.606176    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49c25-predictor-5c44655c88-2xw8x"]
Jul 28 16:42:20.608044 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.608023    2556 scope.go:117] "RemoveContainer" containerID="b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"
Jul 28 16:42:20.616801 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.616780    2556 scope.go:117] "RemoveContainer" containerID="58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"
Jul 28 16:42:20.617113 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:42:20.617087    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59\": container with ID starting with 58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59 not found: ID does not exist" containerID="58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"
Jul 28 16:42:20.617225 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.617120    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59"} err="failed to get container status \"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59\": rpc error: code = NotFound desc = could not find container \"58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59\": container with ID starting with 58d8b48e673f4ebbad61549074d293ef5f898edee8c5874f254c34c8645a0e59 not found: ID does not exist"
Jul 28 16:42:20.617225 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.617140    2556 scope.go:117] "RemoveContainer" containerID="b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"
Jul 28 16:42:20.617410 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:42:20.617390    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58\": container with ID starting with b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58 not found: ID does not exist" containerID="b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"
Jul 28 16:42:20.617462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.617415    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58"} err="failed to get container status \"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58\": rpc error: code = NotFound desc = could not find container \"b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58\": container with ID starting with b6956303feb9a94e1e0be60628d85579314a0798c500af53d4ba45bafbe19a58 not found: ID does not exist"
Jul 28 16:42:20.620692 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.620668    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:42:20.624253 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:20.624233    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49c25-predictor-75f8fbbd8f-bsnmg"]
Jul 28 16:42:21.463324 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:21.463281    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" path="/var/lib/kubelet/pods/93032262-4d31-45cb-9ad2-161cab3accbd/volumes"
Jul 28 16:42:21.463806 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:42:21.463786    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" path="/var/lib/kubelet/pods/addebf89-052d-4d13-9c5c-0fef273bb273/volumes"
Jul 28 16:46:15.559403 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:46:15.559290    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:46:15.565984 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:46:15.565965    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:50:00.867054 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.867013    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:50:00.867645 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.867273    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container" containerID="cri-o://751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1" gracePeriod=30
Jul 28 16:50:00.867645 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.867301    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kube-rbac-proxy" containerID="cri-o://fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4" gracePeriod=30
Jul 28 16:50:00.929343 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.929303    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:50:00.929609 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.929565    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container" containerID="cri-o://c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5" gracePeriod=30
Jul 28 16:50:00.929719 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:00.929607    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kube-rbac-proxy" containerID="cri-o://a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59" gracePeriod=30
Jul 28 16:50:01.731355 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:01.731321    2556 generic.go:356] "Generic (PLEG): container finished" podID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerID="fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4" exitCode=2
Jul 28 16:50:01.731546 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:01.731389    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerDied","Data":"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"}
Jul 28 16:50:01.733330 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:01.733302    2556 generic.go:356] "Generic (PLEG): container finished" podID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerID="a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59" exitCode=2
Jul 28 16:50:01.733462 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:01.733370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerDied","Data":"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"}
Jul 28 16:50:03.739670 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:03.739606    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 16:50:04.355662 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.355607    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:50:04.410295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.410206    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config\") pod \"6d71f4da-75b7-4681-8b59-daeff6441f33\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") "
Jul 28 16:50:04.410295 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.410249    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xqj\" (UniqueName: \"kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj\") pod \"6d71f4da-75b7-4681-8b59-daeff6441f33\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") "
Jul 28 16:50:04.410487 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.410313    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls\") pod \"6d71f4da-75b7-4681-8b59-daeff6441f33\" (UID: \"6d71f4da-75b7-4681-8b59-daeff6441f33\") "
Jul 28 16:50:04.410689 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.410658    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d71f4da-75b7-4681-8b59-daeff6441f33" (UID: "6d71f4da-75b7-4681-8b59-daeff6441f33"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:50:04.412634 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.412586    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d71f4da-75b7-4681-8b59-daeff6441f33" (UID: "6d71f4da-75b7-4681-8b59-daeff6441f33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:50:04.412753 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.412654    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj" (OuterVolumeSpecName: "kube-api-access-c4xqj") pod "6d71f4da-75b7-4681-8b59-daeff6441f33" (UID: "6d71f4da-75b7-4681-8b59-daeff6441f33"). InnerVolumeSpecName "kube-api-access-c4xqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:50:04.479406 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.479380    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:50:04.511152 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w98d6\" (UniqueName: \"kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6\") pod \"0756a0df-0845-4c28-ab0c-e213f648fa1a\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") "
Jul 28 16:50:04.511326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511175    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config\") pod \"0756a0df-0845-4c28-ab0c-e213f648fa1a\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") "
Jul 28 16:50:04.511326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511252    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") pod \"0756a0df-0845-4c28-ab0c-e213f648fa1a\" (UID: \"0756a0df-0845-4c28-ab0c-e213f648fa1a\") "
Jul 28 16:50:04.511460 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511412    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d71f4da-75b7-4681-8b59-daeff6441f33-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.511460 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511423    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.511460 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511434    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xqj\" (UniqueName: \"kubernetes.io/projected/6d71f4da-75b7-4681-8b59-daeff6441f33-kube-api-access-c4xqj\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.511614 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.511532    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0756a0df-0845-4c28-ab0c-e213f648fa1a" (UID: "0756a0df-0845-4c28-ab0c-e213f648fa1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 16:50:04.513523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.513493    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6" (OuterVolumeSpecName: "kube-api-access-w98d6") pod "0756a0df-0845-4c28-ab0c-e213f648fa1a" (UID: "0756a0df-0845-4c28-ab0c-e213f648fa1a"). InnerVolumeSpecName "kube-api-access-w98d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 16:50:04.513523 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.513509    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0756a0df-0845-4c28-ab0c-e213f648fa1a" (UID: "0756a0df-0845-4c28-ab0c-e213f648fa1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 16:50:04.612055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.611994    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0756a0df-0845-4c28-ab0c-e213f648fa1a-proxy-tls\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.612055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.612050    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98d6\" (UniqueName: \"kubernetes.io/projected/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-api-access-w98d6\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.612055 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.612060    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0756a0df-0845-4c28-ab0c-e213f648fa1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-124.ec2.internal\" DevicePath \"\""
Jul 28 16:50:04.749992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.749902    2556 generic.go:356] "Generic (PLEG): container finished" podID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerID="751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1" exitCode=0
Jul 28 16:50:04.749992 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.749981    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"
Jul 28 16:50:04.750519 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.749986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerDied","Data":"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"}
Jul 28 16:50:04.750519 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.750045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p" event={"ID":"6d71f4da-75b7-4681-8b59-daeff6441f33","Type":"ContainerDied","Data":"ccb76fab9f75d93328c2e8d2b1bd30d32fc3ccb4e80469faa6c449fb3e8c1575"}
Jul 28 16:50:04.750519 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.750070    2556 scope.go:117] "RemoveContainer" containerID="fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"
Jul 28 16:50:04.751819 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.751792    2556 generic.go:356] "Generic (PLEG): container finished" podID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerID="c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5" exitCode=0
Jul 28 16:50:04.751940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.751844    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"
Jul 28 16:50:04.751940 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.751872    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerDied","Data":"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"}
Jul 28 16:50:04.752033 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.751941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz" event={"ID":"0756a0df-0845-4c28-ab0c-e213f648fa1a","Type":"ContainerDied","Data":"00363aa2a201c12e3f8db6f34af58d0ba209a09861c91893bbba489f85a0ebc5"}
Jul 28 16:50:04.761147 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.761122    2556 scope.go:117] "RemoveContainer" containerID="751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"
Jul 28 16:50:04.769793 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.769768    2556 scope.go:117] "RemoveContainer" containerID="fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"
Jul 28 16:50:04.770102 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:50:04.770081    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4\": container with ID starting with fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4 not found: ID does not exist" containerID="fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"
Jul 28 16:50:04.770178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.770112    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4"} err="failed to get container status \"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4\": rpc error: code = NotFound desc = could not find container \"fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4\": container with ID starting with fd5aa040ff939c5dabda43a38e97b19f4ba4d806308e3687944633877b168fe4 not found: ID does not exist"
Jul 28 16:50:04.770178 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.770129    2556 scope.go:117] "RemoveContainer" containerID="751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"
Jul 28 16:50:04.770378 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:50:04.770362    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1\": container with ID starting with 751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1 not found: ID does not exist" containerID="751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"
Jul 28 16:50:04.770416 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.770384    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1"} err="failed to get container status \"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1\": rpc error: code = NotFound desc = could not find container \"751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1\": container with ID starting with 751742e8a2a19cf1a5b1cfe30bc7bc9ae8bffcfe00cd89afbda0fcc9862a26f1 not found: ID does not exist"
Jul 28 16:50:04.770416 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.770397    2556 scope.go:117] "RemoveContainer" containerID="a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"
Jul 28 16:50:04.773735 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.773710    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:50:04.777818 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.777793    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d3a3d-predictor-777b55b9d9-bc57p"]
Jul 28 16:50:04.779379 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.779359    2556 scope.go:117] "RemoveContainer" containerID="c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"
Jul 28 16:50:04.788326 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.788306    2556 scope.go:117] "RemoveContainer" containerID="a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"
Jul 28 16:50:04.788420 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.788372    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:50:04.788657 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:50:04.788633    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59\": container with ID starting with a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59 not found: ID does not exist" containerID="a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"
Jul 28 16:50:04.788715 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.788670    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59"} err="failed to get container status \"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59\": rpc error: code = NotFound desc = could not find container \"a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59\": container with ID starting with a6517a23ba594c1ed3524f743c015d3242290527b2c7a786e553968e44039d59 not found: ID does not exist"
Jul 28 16:50:04.788715 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.788694    2556 scope.go:117] "RemoveContainer" containerID="c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"
Jul 28 16:50:04.788965 ip-10-0-142-124 kubenswrapper[2556]: E0728 16:50:04.788940    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5\": container with ID starting with c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5 not found: ID does not exist" containerID="c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"
Jul 28 16:50:04.789124 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.788972    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5"} err="failed to get container status \"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5\": rpc error: code = NotFound desc = could not find container \"c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5\": container with ID starting with c3171ec058e3e39aeaaffcb93cf5e11a014cf828738dc9da0c869b5dfff241c5 not found: ID does not exist"
Jul 28 16:50:04.791762 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:04.791739    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d3a3d-predictor-5bd88cbf88-j5dtz"]
Jul 28 16:50:05.462229 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:05.462146    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" path="/var/lib/kubelet/pods/0756a0df-0845-4c28-ab0c-e213f648fa1a/volumes"
Jul 28 16:50:05.462593 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:05.462579    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" path="/var/lib/kubelet/pods/6d71f4da-75b7-4681-8b59-daeff6441f33/volumes"
Jul 28 16:50:33.324412 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324372    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzjbp/must-gather-l74jf"]
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324763    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324774    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324790    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324796    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324807    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324813    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324822    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324827    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324835    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324840    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324845    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324851    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324858    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324863    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324869    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324876    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324886    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kube-rbac-proxy"
Jul 28 16:50:33.324888 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324892    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324899    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324904    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324909    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324914    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324921    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324926    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324976    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324984    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324991    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.324997    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d71f4da-75b7-4681-8b59-daeff6441f33" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325002    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0756a0df-0845-4c28-ab0c-e213f648fa1a" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325008    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325013    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325018    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325024    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="93032262-4d31-45cb-9ad2-161cab3accbd" containerName="kserve-container"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325029    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6b0fb6-302a-4184-b950-ec0c0ed7f4e0" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325034    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="addebf89-052d-4d13-9c5c-0fef273bb273" containerName="kube-rbac-proxy"
Jul 28 16:50:33.325423 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.325039    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abc91bb-7c02-4a45-9f77-54df554fe14e" containerName="kserve-container"
Jul 28 16:50:33.328000 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.327981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.330913 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.330884    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzjbp\"/\"kube-root-ca.crt\""
Jul 28 16:50:33.331064 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.331022    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzjbp\"/\"openshift-service-ca.crt\""
Jul 28 16:50:33.331142 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.331060    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nzjbp\"/\"default-dockercfg-llbc5\""
Jul 28 16:50:33.337508 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.337482    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzjbp/must-gather-l74jf"]
Jul 28 16:50:33.461285 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.461255    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddm4\" (UniqueName: \"kubernetes.io/projected/2675c5ee-fc9d-4d03-82c0-7165319d2353-kube-api-access-mddm4\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.461465 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.461384    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2675c5ee-fc9d-4d03-82c0-7165319d2353-must-gather-output\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.562402 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.562363    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2675c5ee-fc9d-4d03-82c0-7165319d2353-must-gather-output\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.562402 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.562403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddm4\" (UniqueName: \"kubernetes.io/projected/2675c5ee-fc9d-4d03-82c0-7165319d2353-kube-api-access-mddm4\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.562801 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.562776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2675c5ee-fc9d-4d03-82c0-7165319d2353-must-gather-output\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.571380 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.571355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddm4\" (UniqueName: \"kubernetes.io/projected/2675c5ee-fc9d-4d03-82c0-7165319d2353-kube-api-access-mddm4\") pod \"must-gather-l74jf\" (UID: \"2675c5ee-fc9d-4d03-82c0-7165319d2353\") " pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.648080 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.648040    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzjbp/must-gather-l74jf"
Jul 28 16:50:33.788583 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.788553    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzjbp/must-gather-l74jf"]
Jul 28 16:50:33.791205 ip-10-0-142-124 kubenswrapper[2556]: W0728 16:50:33.791164    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2675c5ee_fc9d_4d03_82c0_7165319d2353.slice/crio-d1f31d0851efe1db5285ec915bdfb9afdc27331cbb30dcfc13a79099351ac52e WatchSource:0}: Error finding container d1f31d0851efe1db5285ec915bdfb9afdc27331cbb30dcfc13a79099351ac52e: Status 404 returned error can't find the container with id d1f31d0851efe1db5285ec915bdfb9afdc27331cbb30dcfc13a79099351ac52e
Jul 28 16:50:33.793163 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.793142    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 16:50:33.892156 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:33.892117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzjbp/must-gather-l74jf" event={"ID":"2675c5ee-fc9d-4d03-82c0-7165319d2353","Type":"ContainerStarted","Data":"d1f31d0851efe1db5285ec915bdfb9afdc27331cbb30dcfc13a79099351ac52e"}
Jul 28 16:50:34.900424 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:34.900389    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzjbp/must-gather-l74jf" event={"ID":"2675c5ee-fc9d-4d03-82c0-7165319d2353","Type":"ContainerStarted","Data":"ccfdc760c42bd82e00b40a8b30f9fb4ffb2b8b8d23253b2ee4a402779e09ae9d"}
Jul 28 16:50:34.900991 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:34.900964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzjbp/must-gather-l74jf" event={"ID":"2675c5ee-fc9d-4d03-82c0-7165319d2353","Type":"ContainerStarted","Data":"51c35e34b75dc5edc3738667a7023a6cd8a82cc9989d34b3c2916ac4f9c481af"}
Jul 28 16:50:34.916603 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:34.916543    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzjbp/must-gather-l74jf" podStartSLOduration=1.198958138 podStartE2EDuration="1.916510861s" totalImagesPullingTime="717.552723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 16:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:50:34.916266985 +0000 UTC m=+2959.967071061" watchObservedRunningTime="2026-07-28 16:50:34.916510861 +0000 UTC m=+2959.967314935"
Jul 28 16:50:35.965094 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:35.965046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j225x_17c73738-c4bf-40fc-ad8c-7bac3ad2db11/global-pull-secret-syncer/0.log"
Jul 28 16:50:36.116807 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:36.116776    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rcrxp_811710f4-2c68-43dd-a1df-f3cad45012c1/konnectivity-agent/0.log"
Jul 28 16:50:36.234363 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:36.234278    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-124.ec2.internal_834e29422605a4563abd0c2b2da8dece/haproxy/0.log"
Jul 28 16:50:40.549433 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.549352    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/alertmanager/0.log"
Jul 28 16:50:40.574554 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.574519    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/config-reloader/0.log"
Jul 28 16:50:40.603219 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.603185    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/kube-rbac-proxy-web/0.log"
Jul 28 16:50:40.635101 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.635068    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/kube-rbac-proxy/0.log"
Jul 28 16:50:40.663700 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.663670    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/kube-rbac-proxy-metric/0.log"
Jul 28 16:50:40.691852 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.691823    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/prom-label-proxy/0.log"
Jul 28 16:50:40.720547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.720515    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b724ab0f-3647-4b00-a4fe-f8a1d0b0a6b8/init-config-reloader/0.log"
Jul 28 16:50:40.802391 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.802304    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jsffh_512d7939-27c8-435a-a509-d43fa1179b76/kube-state-metrics/0.log"
Jul 28 16:50:40.828808 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.828779    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jsffh_512d7939-27c8-435a-a509-d43fa1179b76/kube-rbac-proxy-main/0.log"
Jul 28 16:50:40.858635 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.858586    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-jsffh_512d7939-27c8-435a-a509-d43fa1179b76/kube-rbac-proxy-self/0.log"
Jul 28 16:50:40.939896 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:40.939857    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-ttvjg_cdc9690c-b4ff-454b-b95c-fdadcde0d85e/monitoring-plugin/0.log"
Jul 28 16:50:41.064001 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.063915    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w746b_4bfcea3d-07d4-435e-80a2-7b3787f35f04/node-exporter/0.log"
Jul 28 16:50:41.099302 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.099271    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w746b_4bfcea3d-07d4-435e-80a2-7b3787f35f04/kube-rbac-proxy/0.log"
Jul 28 16:50:41.143404 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.143377    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w746b_4bfcea3d-07d4-435e-80a2-7b3787f35f04/init-textfile/0.log"
Jul 28 16:50:41.336872 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.336792    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-p25nd_46494030-ddac-40b4-9370-e89ae7b3102e/kube-rbac-proxy-main/0.log"
Jul 28 16:50:41.366950 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.366922    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-p25nd_46494030-ddac-40b4-9370-e89ae7b3102e/kube-rbac-proxy-self/0.log"
Jul 28 16:50:41.403839 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.403803    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-p25nd_46494030-ddac-40b4-9370-e89ae7b3102e/openshift-state-metrics/0.log"
Jul 28 16:50:41.671065 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.671029    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cgcsn_706bc98f-b212-4795-a188-646c7c73a311/prometheus-operator/0.log"
Jul 28 16:50:41.698974 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.698932    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-cgcsn_706bc98f-b212-4795-a188-646c7c73a311/kube-rbac-proxy/0.log"
Jul 28 16:50:41.862225 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.862166    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/thanos-query/0.log"
Jul 28 16:50:41.891369 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.891336    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/kube-rbac-proxy-web/0.log"
Jul 28 16:50:41.918538 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.918505    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/kube-rbac-proxy/0.log"
Jul 28 16:50:41.955903 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.955819    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/prom-label-proxy/0.log"
Jul 28 16:50:41.986131 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:41.986088    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/kube-rbac-proxy-rules/0.log"
Jul 28 16:50:42.020089 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:42.020059    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5bb87b85b4-fdrxg_3b9648d0-e141-4ded-b134-68e8a6a0d655/kube-rbac-proxy-metrics/0.log"
Jul 28 16:50:44.358814 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.358764    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8fc455dfd-qmz4c_1a928158-1ba4-461d-a0e4-e90ef7a2e4ec/console/0.log"
Jul 28 16:50:44.407392 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.407361    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-v5vhn_45254f28-7160-4e38-b5d9-ef3d1aad5181/download-server/0.log"
Jul 28 16:50:44.567451 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.567414    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"]
Jul 28 16:50:44.577143 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.577106    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.582087 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.582052    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"]
Jul 28 16:50:44.675401 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.675316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-lib-modules\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.675401 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.675385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-proc\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.675677 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.675419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9j8\" (UniqueName: \"kubernetes.io/projected/4b0330a1-15bf-43e0-993b-368ab3ac9c82-kube-api-access-2p9j8\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.675677 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.675460    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-sys\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.675677 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.675552    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-podres\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.776832 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-sys\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-podres\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-lib-modules\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776914    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-proc\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776927    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-sys\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9j8\" (UniqueName: \"kubernetes.io/projected/4b0330a1-15bf-43e0-993b-368ab3ac9c82-kube-api-access-2p9j8\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777025 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.776995    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-proc\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777247 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.777042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-podres\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.777247 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.777097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b0330a1-15bf-43e0-993b-368ab3ac9c82-lib-modules\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.787029 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.786961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9j8\" (UniqueName: \"kubernetes.io/projected/4b0330a1-15bf-43e0-993b-368ab3ac9c82-kube-api-access-2p9j8\") pod \"perf-node-gather-daemonset-s5blh\" (UID: \"4b0330a1-15bf-43e0-993b-368ab3ac9c82\") " pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:44.893044 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:44.893007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:45.084542 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:45.084448    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"]
Jul 28 16:50:45.982830 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:45.982788    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh" event={"ID":"4b0330a1-15bf-43e0-993b-368ab3ac9c82","Type":"ContainerStarted","Data":"c1129bdbbe606e8b41697c9d97ba117c3eeed02ed283f8171b83066da1812b68"}
Jul 28 16:50:45.983203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:45.982857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh" event={"ID":"4b0330a1-15bf-43e0-993b-368ab3ac9c82","Type":"ContainerStarted","Data":"b1cc691f6ad1e48ebc1c83eb5268bd9102ef1b97ffc68632969b3de80ddfebab"}
Jul 28 16:50:45.983203 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:45.982935    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:45.991057 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:45.991035    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8c2p_7324c4bf-48ba-47b3-8819-83fde99ff606/dns/0.log"
Jul 28 16:50:46.000593 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:46.000544    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh" podStartSLOduration=2.000530233 podStartE2EDuration="2.000530233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 16:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 16:50:45.998015118 +0000 UTC m=+2971.048819192" watchObservedRunningTime="2026-07-28 16:50:46.000530233 +0000 UTC m=+2971.051334306"
Jul 28 16:50:46.034183 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:46.034150    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8c2p_7324c4bf-48ba-47b3-8819-83fde99ff606/kube-rbac-proxy/0.log"
Jul 28 16:50:46.252225 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:46.252142    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6j4ql_edd4ba9f-f16f-42de-ac26-975c51735d1a/dns-node-resolver/0.log"
Jul 28 16:50:46.891814 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:46.891780    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69jt2_425eac07-3ae4-4c5a-94b6-839025d8d7ed/node-ca/0.log"
Jul 28 16:50:48.252416 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:48.252379    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-62m44_c9ba0c09-63e7-4c9a-b6a4-04ec05b9bc8e/serve-healthcheck-canary/0.log"
Jul 28 16:50:49.009368 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:49.009342    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7jgs_458c263d-9f6a-4188-ac89-0eed7b1a8d7b/kube-rbac-proxy/0.log"
Jul 28 16:50:49.035068 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:49.035044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7jgs_458c263d-9f6a-4188-ac89-0eed7b1a8d7b/exporter/0.log"
Jul 28 16:50:49.063756 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:49.063725    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q7jgs_458c263d-9f6a-4188-ac89-0eed7b1a8d7b/extractor/0.log"
Jul 28 16:50:51.351258 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:51.351213    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66f44fdc75-z98l2_73e741fd-312a-4ea3-b9f6-1dfaeae3e078/manager/0.log"
Jul 28 16:50:51.410243 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:51.410214    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-lcpnj_d9a93b2d-ea04-4781-8db0-eec6bb30f23c/server/0.log"
Jul 28 16:50:51.521445 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:51.521417    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-zjmc9_1c0a3517-fecd-480c-b8d6-772d91de5d35/manager/0.log"
Jul 28 16:50:51.805942 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:51.805839    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wtfmq_2e991041-0b7c-40e2-9ad2-5006258e625d/seaweedfs/0.log"
Jul 28 16:50:51.998547 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:51.998515    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nzjbp/perf-node-gather-daemonset-s5blh"
Jul 28 16:50:57.719999 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.719965    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/kube-multus-additional-cni-plugins/0.log"
Jul 28 16:50:57.749774 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.749739    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/egress-router-binary-copy/0.log"
Jul 28 16:50:57.776845 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.776811    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/cni-plugins/0.log"
Jul 28 16:50:57.805394 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.805366    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/bond-cni-plugin/0.log"
Jul 28 16:50:57.835568 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.835526    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/routeoverride-cni/0.log"
Jul 28 16:50:57.863133 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.863106    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/whereabouts-cni-bincopy/0.log"
Jul 28 16:50:57.893660 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:57.893613    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w544q_28ef4649-bfed-43a0-aabc-301174665b34/whereabouts-cni/0.log"
Jul 28 16:50:58.189202 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:58.189169    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fshgz_df7f31c1-0c68-4404-bdf1-e4be1580385f/kube-multus/0.log"
Jul 28 16:50:58.365344 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:58.365307    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2hxs_9da7f48a-38cf-4878-bb8d-53c58670b5d7/network-metrics-daemon/0.log"
Jul 28 16:50:58.391234 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:58.391196    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2hxs_9da7f48a-38cf-4878-bb8d-53c58670b5d7/kube-rbac-proxy/0.log"
Jul 28 16:50:59.729433 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.729404    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-controller/0.log"
Jul 28 16:50:59.753231 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.753189    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/1.log"
Jul 28 16:50:59.753842 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.753818    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovn-acl-logging/0.log"
Jul 28 16:50:59.777274 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.777238    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/kube-rbac-proxy-node/0.log"
Jul 28 16:50:59.803661 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.803615    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 16:50:59.830321 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.830274    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/northd/0.log"
Jul 28 16:50:59.856440 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.856412    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/nbdb/0.log"
Jul 28 16:50:59.883797 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.883765    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/sbdb/0.log"
Jul 28 16:50:59.964080 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:50:59.964045    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mlq9q_7901a8b4-34c9-4668-b0f9-759fff4e035f/ovnkube-controller/0.log"
Jul 28 16:51:01.062303 ip-10-0-142-124 kubenswrapper[2556]: I0728 16:51:01.062274    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-bfrcj_a4e33799-9ac4-4628-a0fb-170ceda351bb/network-check-target-container/0.log"