Sep 09 19:53:10.663752 ip-10-0-136-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 19:53:10.663763 ip-10-0-136-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 19:53:10.663772 ip-10-0-136-11 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 19:53:10.663968 ip-10-0-136-11 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 19:53:20.798771 ip-10-0-136-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 19:53:20.798791 ip-10-0-136-11 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4d86154e64eb46c4a3d3ceea13f2065c --
Sep 09 19:55:00.310608 ip-10-0-136-11 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:55:00.716168 ip-10-0-136-11 kubenswrapper[2547]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:00.716168 ip-10-0-136-11 kubenswrapper[2547]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:55:00.716168 ip-10-0-136-11 kubenswrapper[2547]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:00.716168 ip-10-0-136-11 kubenswrapper[2547]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:55:00.716168 ip-10-0-136-11 kubenswrapper[2547]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:55:00.717830 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.717731    2547 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:55:00.722104 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722087    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.722104 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722103    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.722104 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722107    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722111    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722114    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722117    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722121    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722123    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722126    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722129    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722133    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722137    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722139    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722142    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722145    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722147    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722150    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722153    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722156    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722159    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722161    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722164    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.722211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722167    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722169    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722174    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722178    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722182    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722184    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722187    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722190    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722192    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722195    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722197    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722200    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722203    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722206    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722210    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722213    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722216    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722219    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722222    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722225    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.722709 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722228    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722231    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722234    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722236    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722239    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722243    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722246    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722248    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722251    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722254    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722257    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722259    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722262    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722265    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722267    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722270    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722273    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722275    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722278    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722281    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.723217 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722283    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722299    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722302    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722305    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722307    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722310    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722313    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722316    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722319    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722322    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722325    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722328    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722330    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722333    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722336    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722338    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722341    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722344    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722346    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722349    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.723723 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722351    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722355    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722359    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722363    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722366    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722370    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722373    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722376    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722379    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722382    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722385    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722388    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722390    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722393    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722396    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722399    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.722402    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723747    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723756    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.724207 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723759    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723762    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723765    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723769    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723772    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723775    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723777    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723780    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723784    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723788    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723791    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723794    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723797    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723800    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723803    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723806    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723810    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723813    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723816    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723818    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.724682 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723821    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723824    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723827    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723830    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723832    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723835    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723838    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723841    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723844    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723846    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723849    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723852    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723854    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723857    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723861    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723864    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723868    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723872    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723876    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.725211 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723879    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723882    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723884    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723887    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723889    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723892    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723894    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723897    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723900    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723903    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723905    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723908    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723911    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723914    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723916    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723919    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723922    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723924    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723926    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.725744 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723930    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723933    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723936    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723939    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723942    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723945    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723947    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723950    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723952    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723955    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723958    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723961    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723964    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723966    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723969    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723971    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723974    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723976    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723979    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723981    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.726231 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723985    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723990    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723993    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723995    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.723998    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724000    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724003    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724005    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724008    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724011    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724013    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724016    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724019    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724022    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724025    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724027    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724030    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724032    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724035    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724427    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.726748 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724432    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724436    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724439    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724442    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724445    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724449    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724451    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724454    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724457    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724459    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724462    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724465    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724468    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724471    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724474    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724477    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724480    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724483    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724485    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724488    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.727244 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724491    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724494    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724496    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724499    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724502    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724506    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724510    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724513    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724516    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724519    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724521    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724524    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724527    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724530    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724532    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724535    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724538    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724541    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724545    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.727743 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724548    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724551    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724554    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724557    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724559    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724562    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724565    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724567    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724570    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724573    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724575    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724578    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724581    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724584    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724586    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724589    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724592    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724594    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724597    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724600    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.728302 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724602    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724605    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724608    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724611    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724613    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724616    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724618    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724621    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724623    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724626    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724629    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724631    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724636    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724639    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724643    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724646    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724649    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724652    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724655    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.728803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724658    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724661    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724664    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724666    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724669    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724672    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724674    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724677    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724680    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724682    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724685    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724688    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724691    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724693    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724696    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724699    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724702    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724704    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724707    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.724709    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.729269 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724794    2547 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724824    2547 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724833    2547 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724838    2547 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724843    2547 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724847    2547 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724851    2547 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724855    2547 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724859    2547 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724861    2547 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724865    2547 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724868    2547 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724872    2547 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724875    2547 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724878    2547 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724881    2547 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724884    2547 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724887    2547 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724891    2547 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724894    2547 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724897    2547 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724900    2547 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724904    2547 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724908    2547 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:55:00.729781 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724910    2547 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724913    2547 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724917    2547 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724920    2547 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724923    2547 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724926    2547 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724929    2547 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724932    2547 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724936    2547 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724940    2547 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724943    2547 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724945    2547 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724948    2547 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724952    2547 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724956    2547 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724960    2547 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724963    2547 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724966    2547 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724968    2547 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724975    2547 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724978    2547 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724981    2547 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724984    2547 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724986    2547 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724989    2547 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:55:00.730401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724992    2547 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724995    2547 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.724998    2547 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725001    2547 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725004    2547 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725008    2547 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725011    2547 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725014    2547 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725017    2547 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725022    2547 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725025    2547 flags.go:64] FLAG: --help="false"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725028    2547 flags.go:64] FLAG: --hostname-override="ip-10-0-136-11.ec2.internal"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725031    2547 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725034    2547 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725037    2547 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725041    2547 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725044    2547 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725047    2547 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725050    2547 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725052    2547 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725055    2547 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725059    2547 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725062    2547 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725065    2547 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:55:00.731058 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725068    2547 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725071    2547 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725076    2547 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725078    2547 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725081    2547 flags.go:64] FLAG: --lock-file=""
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725084    2547 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725087    2547 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725090    2547 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725096    2547 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725099    2547 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725102    2547 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725105    2547 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725108    2547 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725111    2547 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725114    2547 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725117    2547 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725125    2547 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725130    2547 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725134    2547 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725137    2547 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725140    2547 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725142    2547 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725145    2547 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725148    2547 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725151    2547 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:55:00.731690 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725154    2547 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725162    2547 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725165    2547 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725168    2547 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725171    2547 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725174    2547 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725181    2547 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725184    2547 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725187    2547 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725190    2547 flags.go:64] FLAG: --port="10250"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725194    2547 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725197    2547 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cad29299f6f18762"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725200    2547 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725203    2547 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725206    2547 flags.go:64] FLAG: --register-node="true"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725209    2547 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725213    2547 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725216    2547 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725220    2547 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725223    2547 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725227    2547 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725230    2547 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725233    2547 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725236    2547 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725240    2547 flags.go:64] FLAG: --runonce="false"
Sep 09 19:55:00.732340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725243    2547 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725246    2547 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725249    2547 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725252    2547 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725255    2547 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725258    2547 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725261    2547 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725265    2547 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725268    2547 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725270    2547 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725273    2547 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725277    2547 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725280    2547 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725284    2547 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725300    2547 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725305    2547 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725308    2547 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725313    2547 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725317    2547 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725320    2547 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725323    2547 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725327    2547 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725329    2547 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725332    2547 flags.go:64] FLAG: --v="2"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725337    2547 flags.go:64] FLAG: --version="false"
Sep 09 19:55:00.732958 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725340    2547 flags.go:64] FLAG: --vmodule=""
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725344    2547 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.725348    2547 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725460    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725464    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725467    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725471    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725474    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725477    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725480    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725483    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725486    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725489    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725491    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725494    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725497    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725499    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725502    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725505    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725508    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.733610 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725511    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725514    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725516    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725519    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725523    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725526    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725529    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725531    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725534    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725536    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725539    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725542    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725544    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725547    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725550    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725552    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725555    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725557    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725561    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725564    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.734121 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725567    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725570    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725572    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725575    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725577    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725592    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725595    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725597    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725600    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725603    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725605    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725608    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725611    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725614    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725617    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725619    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725624    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725627    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725630    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.734675 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725634    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725637    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725640    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725643    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725646    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725650    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725652    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725655    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725658    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725661    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725664    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725670    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725673    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725676    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725679    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725682    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725685    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725688    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725691    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.735149 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725693    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725696    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725699    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725703    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725706    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725709    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725712    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725714    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725717    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725720    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725725    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725728    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725730    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725733    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725736    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725739    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725741    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725744    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725747    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725749    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.735635 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725752    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725755    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725758    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.725761    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.726384    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.733413    2547 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.733429    2547 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.734146    2547 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.734158    2547 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734218    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734223    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734228    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734232    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734235    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.736134 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734238    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734241    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734243    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734246    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734249    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734251    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734254    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734257    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734260    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734263    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734265    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734268    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734270    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734273    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734276    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734278    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734281    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734283    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734299    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734303    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.736518 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734306    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734308    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734311    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734314    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734318    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734321    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734324    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734327    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734330    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734333    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734335    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734338    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734340    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734343    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734346    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734348    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734351    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734354    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734356    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734359    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.737003 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734361    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734364    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734367    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734369    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734373    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734377    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734380    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734383    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734386    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734389    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734392    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734395    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734397    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734400    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734404    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734408    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734412    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734415    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734418    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.737653 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734421    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734424    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734427    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734430    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734433    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734436    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734440    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734444    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734446    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734449    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734452    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734455    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734457    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734460    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734462    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734464    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734467    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734470    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734472    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.738242 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734475    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734477    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734480    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734483    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734485    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734488    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734490    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734493    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734496    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734498    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734501    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734504    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734507    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734509    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734512    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.739053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734515    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.734520    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734632    2547 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734639    2547 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734643    2547 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734646    2547 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734649    2547 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734652    2547 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734655    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734658    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734661    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734663    2547 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734666    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734669    2547 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:55:00.739470 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734671    2547 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734674    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734677    2547 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734679    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734682    2547 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734685    2547 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734688    2547 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734692    2547 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734695    2547 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734698    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734701    2547 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734705    2547 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734708    2547 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734710    2547 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734713    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734716    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734719    2547 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734722    2547 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734724    2547 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:55:00.739819 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734727    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734730    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734732    2547 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734735    2547 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734738    2547 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734740    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734743    2547 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734745    2547 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734748    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734751    2547 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734754    2547 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734756    2547 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734759    2547 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734762    2547 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734764    2547 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734767    2547 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734769    2547 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734772    2547 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734774    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734778    2547 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:55:00.740307 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734780    2547 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734783    2547 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734786    2547 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734789    2547 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734791    2547 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734794    2547 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734796    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734799    2547 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734801    2547 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734804    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734807    2547 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734810    2547 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734812    2547 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734815    2547 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734817    2547 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734820    2547 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734822    2547 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734825    2547 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734829    2547 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734832    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:55:00.740814 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734835    2547 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734838    2547 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734840    2547 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734843    2547 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734846    2547 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734848    2547 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734851    2547 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734854    2547 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734856    2547 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734859    2547 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734861    2547 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734864    2547 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734867    2547 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734869    2547 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734872    2547 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734875    2547 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734877    2547 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734880    2547 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734883    2547 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734885    2547 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:55:00.741313 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734888    2547 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734891    2547 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734893    2547 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734896    2547 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734899    2547 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734901    2547 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734904    2547 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:00.734906    2547 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.734911    2547 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.735023    2547 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.736986    2547 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.738480    2547 server.go:1019] "Starting client certificate rotation"
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.738742    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:00.741803 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.738808    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:55:00.742145 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.741819    2547 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:00.743081 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.743064    2547 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:55:00.752195 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.752178    2547 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:55:00.755399 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.755381    2547 log.go:25] "Validated CRI v1 image API"
Sep 09 19:55:00.756325 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.756303    2547 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:55:00.762659 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.762636    2547 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bd67c20e-314a-4458-9924-11104ca6c598:/dev/nvme0n1p4 f3a45c15-47e5-4eed-afc6-9270b0d803bf:/dev/nvme0n1p3]
Sep 09 19:55:00.762746 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.762657    2547 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:55:00.767361 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.767344    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:00.768819 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.768705    2547 manager.go:216] Machine: {Timestamp:2026-09-09 19:55:00.766369536 +0000 UTC m=+0.337302261 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114508 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebbaa4cb7cbf2e587b95d7ae808ac SystemUUID:ec2ebbaa-4cb7-cbf2-e587-b95d7ae808ac BootID:4d86154e-64eb-46c4-a3d3-ceea13f2065c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bf:9d:3c:94:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bf:9d:3c:94:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:5a:6f:7e:9b:a1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:55:00.768819 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.768811    2547 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:55:00.768925 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.768900    2547 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:55:00.771747 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.771720    2547 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:55:00.771897 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.771750    2547 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:55:00.772615 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.772605    2547 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:55:00.772653 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.772618    2547 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:55:00.772653 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.772627    2547 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:00.773244 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.773234    2547 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:55:00.773975 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.773965    2547 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:55:00.777623 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777611    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:00.777730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777721    2547 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:55:00.777775 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777749    2547 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:55:00.777775 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777766    2547 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:55:00.777856 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777795    2547 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:55:00.777856 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777830    2547 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:55:00.777856 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.777846    2547 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:55:00.778385 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.778346    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:00.778385 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.778377    2547 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:55:00.783493 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.783417    2547 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:55:00.786215 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.786197    2547 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:55:00.786307 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.786230    2547 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:55:00.787726 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787714    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787731    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787738    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787744    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787753    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787758    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:55:00.787772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787771    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:55:00.787928 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787777    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:55:00.787928 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787785    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:55:00.787928 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787791    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:55:00.787928 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787800    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:55:00.787928 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.787805    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:55:00.788540 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.788526    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:55:00.788578 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.788545    2547 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:55:00.791069 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.791037    2547 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:55:00.791173 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.791153    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:55:00.791323 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.791302    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:55:00.792471 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.792459    2547 server.go:1268] "Started kubelet"
Sep 09 19:55:00.792578 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.792543    2547 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:55:00.792632 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.792555    2547 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:55:00.792632 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.792625    2547 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:55:00.793485 ip-10-0-136-11 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:55:00.794517 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.794496    2547 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:55:00.795365 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.794709    2547 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:55:00.800528 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.800503    2547 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:55:00.803483 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.803468    2547 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:00.803991 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.803972    2547 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:55:00.804193 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.804168    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:00.804573 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.804554    2547 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:55:00.804573 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.804556    2547 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:55:00.804707 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.804580    2547 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:55:00.804707 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.804699    2547 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:55:00.804793 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.804710    2547 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:55:00.805085 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805071    2547 factory.go:153] Registering CRI-O factory
Sep 09 19:55:00.805122 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805088    2547 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:55:00.805150 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805128    2547 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:55:00.805150 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805135    2547 factory.go:55] Registering systemd factory
Sep 09 19:55:00.805150 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805140    2547 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:55:00.805243 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805152    2547 factory.go:103] Registering Raw factory
Sep 09 19:55:00.805243 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.805170    2547 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:55:00.805850 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.805827    2547 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:55:00.806008 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.805991    2547 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:55:00.806531 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.806515    2547 manager.go:349] Starting recovery of all containers
Sep 09 19:55:00.806915 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.805812    2547 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-11.ec2.internal.18d3bf328ab251f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-11.ec2.internal,UID:ip-10-0-136-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-11.ec2.internal,},FirstTimestamp:2026-09-09 19:55:00.792431096 +0000 UTC m=+0.363363801,LastTimestamp:2026-09-09 19:55:00.792431096 +0000 UTC m=+0.363363801,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-11.ec2.internal,}"
Sep 09 19:55:00.817965 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.817939    2547 manager.go:354] Recovery completed
Sep 09 19:55:00.818633 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.818609    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jx2jr"
Sep 09 19:55:00.820092 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.820044    2547 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 19:55:00.823585 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.823570    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:00.825386 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.825368    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:00.825464 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.825399    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:00.825464 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.825409    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:00.826002 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.825988    2547 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:55:00.826066 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826009    2547 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:55:00.826066 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826024    2547 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:55:00.826066 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826060    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:00.826303 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826277    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jx2jr"
Sep 09 19:55:00.826819 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826805    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:00.826885 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826833    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:00.826885 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.826846    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:00.828303 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.828275    2547 policy_none.go:49] "None policy: Start"
Sep 09 19:55:00.828303 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.828305    2547 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:55:00.828445 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.828318    2547 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:55:00.830252 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.830240    2547 policy_none.go:47] "Start"
Sep 09 19:55:00.858966 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.858947    2547 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.858994    2547 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859008    2547 server.go:87] "Starting device plugin registration server"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859386    2547 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859400    2547 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859570    2547 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859664    2547 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.859673    2547 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.860173    2547 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:55:00.882909 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.860222    2547 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:00.939432 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.939392    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:55:00.940602 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.940587    2547 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:55:00.940672 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.940638    2547 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:55:00.940672 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.940662    2547 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:55:00.940774 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.940749    2547 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:55:00.943474 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.943454    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:00.960487 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.960470    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:00.961603 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.961586    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:00.961670 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.961617    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:00.961670 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.961628    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:00.961670 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.961652    2547 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-11.ec2.internal"
Sep 09 19:55:00.970985 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:00.970922    2547 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-11.ec2.internal"
Sep 09 19:55:00.970985 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.970943    2547 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-11.ec2.internal\": node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:00.997716 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:00.997696    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.041111 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.041036    2547 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"]
Sep 09 19:55:01.041230 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.041158    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:01.042071 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.042047    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:01.042164 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.042083    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:01.042164 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.042099    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:01.044126 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044113    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:01.044244 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044228    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.044300 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044265    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:01.044835 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044817    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:01.044915 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044847    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:01.044915 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044857    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:01.044915 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044819    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:01.045060 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044924    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:01.045060 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.044941    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:01.046893 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.046877    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.046972 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.046900    2547 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:55:01.047577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.047559    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:55:01.047647 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.047593    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:55:01.047647 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.047610    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:55:01.066944 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.066925    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-11.ec2.internal\" not found" node="ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.071340 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.071325    2547 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-11.ec2.internal\" not found" node="ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.098117 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.098098    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.106766 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.106746    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.106831 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.106771    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.106831 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.106787    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a90e5cd9b3bb35e0b18f98e5a2e76f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-11.ec2.internal\" (UID: \"0a90e5cd9b3bb35e0b18f98e5a2e76f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.198360 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.198331    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.207733 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207713    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.207799 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207741    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.207799 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207759    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a90e5cd9b3bb35e0b18f98e5a2e76f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-11.ec2.internal\" (UID: \"0a90e5cd9b3bb35e0b18f98e5a2e76f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.207868 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207805    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.207868 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207812    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0a90e5cd9b3bb35e0b18f98e5a2e76f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-11.ec2.internal\" (UID: \"0a90e5cd9b3bb35e0b18f98e5a2e76f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.207868 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.207819    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3dfaafcae6539e2d64675470358ee9d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal\" (UID: \"e3dfaafcae6539e2d64675470358ee9d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.299213 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.299139    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.372709 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.372664    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.375012 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.374994    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.399449 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.399411    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.506622 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.506573    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.612038 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.611973    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.712756 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.712723    2547 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-11.ec2.internal\" not found"
Sep 09 19:55:01.736667 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.736642    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:01.738735 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.738720    2547 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:55:01.738839 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.738823    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:01.738839 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.738824    2547 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:55:01.753247 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.753220    2547 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:01.777979 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.777950    2547 apiserver.go:52] "Watching apiserver"
Sep 09 19:55:01.786934 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.786914    2547 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:55:01.788130 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.788104    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k","openshift-cluster-node-tuning-operator/tuned-8qhwg","openshift-dns/node-resolver-g2297","openshift-image-registry/node-ca-plptx","openshift-multus/multus-additional-cni-plugins-svpxv","openshift-multus/multus-rx99p","openshift-network-diagnostics/network-check-target-cfnf2","openshift-network-operator/iptables-alerter-7zmp2","kube-system/global-pull-secret-syncer-sm8vh","kube-system/konnectivity-agent-rsnjr","openshift-multus/network-metrics-daemon-q9n5s","openshift-ovn-kubernetes/ovnkube-node-26bsm"]
Sep 09 19:55:01.790753 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.790738    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.792718 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.792698    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:55:01.792993 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.792981    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.793517 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.793494    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lr57\""
Sep 09 19:55:01.793625 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.793541    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.793625 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.793501    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.794897 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.794878    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.795564 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.795545    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.795716 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.795694    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k4v5g\""
Sep 09 19:55:01.796946 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.796928    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.798614 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.798591    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.798842 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.798827    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l8k4n\""
Sep 09 19:55:01.798842 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.798837    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.799211 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.799198    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.801018 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.800994    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.801018 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.801006    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pc922\""
Sep 09 19:55:01.801185 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.801007    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.801185 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.801010    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:55:01.801380 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.801350    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.803122 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803105    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:55:01.803213 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803175    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.803494 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803481    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.803552 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803543    2547 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:55:01.803812 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803793    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:55:01.803891 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803825    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6dhsg\""
Sep 09 19:55:01.803891 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.803882    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:55:01.804075 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.804050    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.804376 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.804139    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.806096 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.806079    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2jq8f\""
Sep 09 19:55:01.806190 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.806151    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:55:01.806250 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.806228    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:01.806326 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.806293    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:01.808689 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.808669    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:01.810446 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.810430    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.810658 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.810644    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.810763 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.810741    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlkmr\""
Sep 09 19:55:01.810983 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.810968    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:55:01.811106 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.811088    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:01.812891 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.812869    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.812993 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.812902    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6sq\" (UniqueName: \"kubernetes.io/projected/a70313ac-da9c-441e-89bb-da79eb54fdb8-kube-api-access-hp6sq\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.812993 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.812930    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-conf-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.812993 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.812955    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-etc-kubernetes\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.812993 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.812979    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-conf\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813049    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-system-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813052    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813107    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-device-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813139    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysconfig\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813182    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bh682\""
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813191    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:55:01.813204 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813164    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-cnibin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813225    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-multus\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813252    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-sys-fs\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813276    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-tmp\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813318    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163aa480-c64c-49e6-aa38-b51aabaf7eb8-serviceca\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813364    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-cnibin\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813418    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-cni-binary-copy\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813444    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-kubelet\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813468    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813488    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-host\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813510    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163aa480-c64c-49e6-aa38-b51aabaf7eb8-host\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.813604 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813555    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-system-cni-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813643    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813601    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813689    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-k8s-cni-cncf-io\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.813709    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813720    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-registration-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813747    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-kubernetes\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813778    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-var-lib-kubelet\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813803    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtl9\" (UniqueName: \"kubernetes.io/projected/77b50dad-2bbc-42ab-8948-b921b410728a-kube-api-access-bmtl9\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813843    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kj5\" (UniqueName: \"kubernetes.io/projected/163aa480-c64c-49e6-aa38-b51aabaf7eb8-kube-api-access-h5kj5\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813938    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-socket-dir-parent\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.813986    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-bin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814010    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-hostroot\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814061    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47457\" (UniqueName: \"kubernetes.io/projected/32bab1db-f8ce-4f25-8b38-c48cfae58857-kube-api-access-47457\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814091    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlgvx\" (UniqueName: \"kubernetes.io/projected/0c874116-2f48-4afd-92dd-caf462ac9684-kube-api-access-hlgvx\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.814141 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814119    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-systemd\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814159    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-run\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814204    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-lib-modules\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814234    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814259    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c874116-2f48-4afd-92dd-caf462ac9684-hosts-file\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814282    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c874116-2f48-4afd-92dd-caf462ac9684-tmp-dir\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814305    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814351    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf7gx\" (UniqueName: \"kubernetes.io/projected/35796d71-fc30-4fe5-8b84-6355b621efff-kube-api-access-bf7gx\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814371    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-modprobe-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814396    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-sys\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814418    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-etc-tuned\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814435    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814450    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814464    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-netns\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814486    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-daemon-config\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814516    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-multus-certs\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.814730 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814546    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.815281 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814564    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-socket-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.815281 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814585    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.815281 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.814952    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:01.815281 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.815015    2547 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"
Sep 09 19:55:01.816555 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.816533    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.818526 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.818506    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p59rt\""
Sep 09 19:55:01.819031 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819008    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal"]
Sep 09 19:55:01.819128 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.818999    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:55:01.819128 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819051    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:55:01.819210 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819129    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:01.819210 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.819189    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:01.819364 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819344    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:55:01.819437 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819372    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:55:01.819437 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819404    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:55:01.819437 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819405    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:55:01.819650 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.819633    2547 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:55:01.828207 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.828187    2547 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:55:01.828335 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.828319    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal"]
Sep 09 19:55:01.831456 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.831426    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:50:01 +0000 UTC" deadline="2028-03-04 23:49:44.620490532 +0000 UTC"
Sep 09 19:55:01.831549 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.831456    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13011h54m42.789039184s"
Sep 09 19:55:01.843936 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.843918    2547 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmq5c"
Sep 09 19:55:01.853110 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.853090    2547 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmq5c"
Sep 09 19:55:01.905299 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.905277    2547 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:55:01.915127 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915102    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.915257 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915140    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-conf\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.915257 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915179    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-var-lib-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.915257 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915210    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-config\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.915257 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915236    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpz7\" (UniqueName: \"kubernetes.io/projected/90d331aa-5052-4a95-91cc-b82896d0d9df-kube-api-access-5zpz7\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:01.915257 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915239    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915263    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915313    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-conf-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915341    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16188a2b-afac-4c06-a671-f786c0937a3e-host-slash\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915363    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-bin\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915380    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-netd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915398    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-system-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915415    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-device-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915439    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-slash\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915443    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-conf\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915492    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-conf-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.915512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915497    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-ovn\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915542    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29ee8571-51bc-4fee-a100-6f72cb11057a-ovn-node-metrics-cert\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915592    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-device-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915609    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-script-lib\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915674    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-cnibin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915709    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-tmp\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915720    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-cnibin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915735    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163aa480-c64c-49e6-aa38-b51aabaf7eb8-serviceca\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915768    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-kubelet-config\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915770    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-system-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915891    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-systemd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916062 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.915962    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916041    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-konnectivity-ca\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916093    2547 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916130    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-cnibin\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916163    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-host\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916211    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-host\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916240    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-cnibin\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916244    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-log-socket\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916286    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-system-cni-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916334    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-var-lib-kubelet\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916357    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-netns\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916373    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-system-cni-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916383    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-hostroot\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916399    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-var-lib-kubelet\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916415    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-systemd\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916419    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-hostroot\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916434    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-run\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916441    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.916577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916456    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-lib-modules\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916464    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-systemd\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916476    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16188a2b-afac-4c06-a671-f786c0937a3e-iptables-alerter-script\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916497    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-systemd-units\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916508    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-run\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916523    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916547    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-agent-certs\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916557    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-lib-modules\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916564    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf7gx\" (UniqueName: \"kubernetes.io/projected/35796d71-fc30-4fe5-8b84-6355b621efff-kube-api-access-bf7gx\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-modprobe-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916616    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-etc-tuned\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916639    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-kubelet\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916701    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916706    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-modprobe-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916754    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-multus-certs\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916786    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-cni-dir\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916784    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/163aa480-c64c-49e6-aa38-b51aabaf7eb8-serviceca\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.917480 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916854    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-multus-certs\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916880    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-socket-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916912    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916945    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.916975    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6sq\" (UniqueName: \"kubernetes.io/projected/a70313ac-da9c-441e-89bb-da79eb54fdb8-kube-api-access-hp6sq\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917002    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-etc-kubernetes\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917028    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94hmn\" (UniqueName: \"kubernetes.io/projected/16188a2b-afac-4c06-a671-f786c0937a3e-kube-api-access-94hmn\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917056    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysconfig\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917080    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-dbus\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917075    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-socket-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917108    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-multus\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917133    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-sys-fs\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917148    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-etc-selinux\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917161    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-etc-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917187    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-node-log\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917214    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-cni-binary-copy\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917215    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysconfig\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.918342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917254    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-etc-kubernetes\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917281    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-kubelet\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917286    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-sys-fs\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917336    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-multus\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917375    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-kubelet\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917415    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163aa480-c64c-49e6-aa38-b51aabaf7eb8-host\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917470    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnck\" (UniqueName: \"kubernetes.io/projected/29ee8571-51bc-4fee-a100-6f72cb11057a-kube-api-access-xpnck\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917502    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-sysctl-d\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917503    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917545    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-k8s-cni-cncf-io\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917574    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-registration-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917615    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-kubernetes\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917644    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtl9\" (UniqueName: \"kubernetes.io/projected/77b50dad-2bbc-42ab-8948-b921b410728a-kube-api-access-bmtl9\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917670    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kj5\" (UniqueName: \"kubernetes.io/projected/163aa480-c64c-49e6-aa38-b51aabaf7eb8-kube-api-access-h5kj5\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917698    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-env-overrides\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917746    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-socket-dir-parent\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919255 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917846    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-bin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917851    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-cni-binary-copy\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917876    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47457\" (UniqueName: \"kubernetes.io/projected/32bab1db-f8ce-4f25-8b38-c48cfae58857-kube-api-access-47457\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917904    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlgvx\" (UniqueName: \"kubernetes.io/projected/0c874116-2f48-4afd-92dd-caf462ac9684-kube-api-access-hlgvx\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917932    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917980    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918010    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c874116-2f48-4afd-92dd-caf462ac9684-hosts-file\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917906    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/163aa480-c64c-49e6-aa38-b51aabaf7eb8-host\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.917934    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918080    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c874116-2f48-4afd-92dd-caf462ac9684-tmp-dir\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918096    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-k8s-cni-cncf-io\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918110    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918128    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-sys\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918147    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918164    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-netns\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918190    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c874116-2f48-4afd-92dd-caf462ac9684-hosts-file\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918146    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-etc-kubernetes\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.919939 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918190    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-socket-dir-parent\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918227    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-daemon-config\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918228    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-run-netns\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918253    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b50dad-2bbc-42ab-8948-b921b410728a-sys\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918292    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32bab1db-f8ce-4f25-8b38-c48cfae58857-host-var-lib-cni-bin\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918352    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70313ac-da9c-441e-89bb-da79eb54fdb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918571    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a70313ac-da9c-441e-89bb-da79eb54fdb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918607    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c874116-2f48-4afd-92dd-caf462ac9684-tmp-dir\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918729    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35796d71-fc30-4fe5-8b84-6355b621efff-registration-dir\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.918758    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32bab1db-f8ce-4f25-8b38-c48cfae58857-multus-daemon-config\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.920047    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-tmp\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.920818 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.920102    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b50dad-2bbc-42ab-8948-b921b410728a-etc-tuned\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.926420 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.926398    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf7gx\" (UniqueName: \"kubernetes.io/projected/35796d71-fc30-4fe5-8b84-6355b621efff-kube-api-access-bf7gx\") pod \"aws-ebs-csi-driver-node-gb82k\" (UID: \"35796d71-fc30-4fe5-8b84-6355b621efff\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:01.926511 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.926483    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6sq\" (UniqueName: \"kubernetes.io/projected/a70313ac-da9c-441e-89bb-da79eb54fdb8-kube-api-access-hp6sq\") pod \"multus-additional-cni-plugins-svpxv\" (UID: \"a70313ac-da9c-441e-89bb-da79eb54fdb8\") " pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:01.926511 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.926497    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtl9\" (UniqueName: \"kubernetes.io/projected/77b50dad-2bbc-42ab-8948-b921b410728a-kube-api-access-bmtl9\") pod \"tuned-8qhwg\" (UID: \"77b50dad-2bbc-42ab-8948-b921b410728a\") " pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:01.926795 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.926778    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47457\" (UniqueName: \"kubernetes.io/projected/32bab1db-f8ce-4f25-8b38-c48cfae58857-kube-api-access-47457\") pod \"multus-rx99p\" (UID: \"32bab1db-f8ce-4f25-8b38-c48cfae58857\") " pod="openshift-multus/multus-rx99p"
Sep 09 19:55:01.927837 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.927817    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:01.927837 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.927840    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:01.928007 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.927854    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.928007 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:01.927926    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:02.42790374 +0000 UTC m=+1.998836450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.928926 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.928893    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kj5\" (UniqueName: \"kubernetes.io/projected/163aa480-c64c-49e6-aa38-b51aabaf7eb8-kube-api-access-h5kj5\") pod \"node-ca-plptx\" (UID: \"163aa480-c64c-49e6-aa38-b51aabaf7eb8\") " pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:01.930090 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:01.930073    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlgvx\" (UniqueName: \"kubernetes.io/projected/0c874116-2f48-4afd-92dd-caf462ac9684-kube-api-access-hlgvx\") pod \"node-resolver-g2297\" (UID: \"0c874116-2f48-4afd-92dd-caf462ac9684\") " pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:02.019090 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019053    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-netns\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019090 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019088    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16188a2b-afac-4c06-a671-f786c0937a3e-iptables-alerter-script\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019105    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-systemd-units\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019121    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019176    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-agent-certs\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-kubelet\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019201    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-netns\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019204    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019213    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019266    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94hmn\" (UniqueName: \"kubernetes.io/projected/16188a2b-afac-4c06-a671-f786c0937a3e-kube-api-access-94hmn\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.019296    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019305    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-dbus\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019288    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-systemd-units\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019327    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-etc-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019350    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-node-log\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.019377    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:02.519354309 +0000 UTC m=+2.090287014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019382    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-node-log\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019396    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-dbus\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.019492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019271    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-kubelet\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019411    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnck\" (UniqueName: \"kubernetes.io/projected/29ee8571-51bc-4fee-a100-6f72cb11057a-kube-api-access-xpnck\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019426    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-etc-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019443    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-env-overrides\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019477    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019533    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-var-lib-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019557    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-config\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019592    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpz7\" (UniqueName: \"kubernetes.io/projected/90d331aa-5052-4a95-91cc-b82896d0d9df-kube-api-access-5zpz7\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.019609    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019616    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-var-lib-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019622    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16188a2b-afac-4c06-a671-f786c0937a3e-host-slash\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019665    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16188a2b-afac-4c06-a671-f786c0937a3e-host-slash\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.019671    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:02.519654006 +0000 UTC m=+2.090586705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019714    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-bin\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019744    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-netd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019761    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-bin\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019772    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-slash\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.020504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019806    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-ovn\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019818    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-cni-netd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019830    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019860    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29ee8571-51bc-4fee-a100-6f72cb11057a-ovn-node-metrics-cert\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019867    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-run-ovn-kubernetes\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019889    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-script-lib\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019888    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/16188a2b-afac-4c06-a671-f786c0937a3e-iptables-alerter-script\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019896    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-ovn\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019916    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-host-slash\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019926    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-kubelet-config\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019955    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-systemd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019986    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.019990    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-env-overrides\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020037    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-systemd\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020035    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-konnectivity-ca\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020052    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-run-openvswitch\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020164    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99c7cf30-f214-4557-87ce-05025c7000eb-kubelet-config\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.021132 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020201    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-log-socket\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020267    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/29ee8571-51bc-4fee-a100-6f72cb11057a-log-socket\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020590    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-config\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.021720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020614    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-konnectivity-ca\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:02.021720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.020636    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/29ee8571-51bc-4fee-a100-6f72cb11057a-ovnkube-script-lib\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.022389 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.022364    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/29ee8571-51bc-4fee-a100-6f72cb11057a-ovn-node-metrics-cert\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.022473 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.022416    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a10dc16-1c01-4fe5-9fff-1003d62c6ab0-agent-certs\") pod \"konnectivity-agent-rsnjr\" (UID: \"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0\") " pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:02.028813 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.028787    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94hmn\" (UniqueName: \"kubernetes.io/projected/16188a2b-afac-4c06-a671-f786c0937a3e-kube-api-access-94hmn\") pod \"iptables-alerter-7zmp2\" (UID: \"16188a2b-afac-4c06-a671-f786c0937a3e\") " pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.028953 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.028929    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnck\" (UniqueName: \"kubernetes.io/projected/29ee8571-51bc-4fee-a100-6f72cb11057a-kube-api-access-xpnck\") pod \"ovnkube-node-26bsm\" (UID: \"29ee8571-51bc-4fee-a100-6f72cb11057a\") " pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.031889 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.031871    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpz7\" (UniqueName: \"kubernetes.io/projected/90d331aa-5052-4a95-91cc-b82896d0d9df-kube-api-access-5zpz7\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:02.112546 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.112492    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k"
Sep 09 19:55:02.121279 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.121205    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8qhwg"
Sep 09 19:55:02.143197 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.143173    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2297"
Sep 09 19:55:02.154772 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.154746    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-plptx"
Sep 09 19:55:02.169197 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.169164    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-svpxv"
Sep 09 19:55:02.187061 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.187034    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rx99p"
Sep 09 19:55:02.199706 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.199663    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c874116_2f48_4afd_92dd_caf462ac9684.slice/crio-83a049d301b4d56e0498e7ac751cb043bf08a5e4982cda97b549d5e6b1aabb05 WatchSource:0}: Error finding container 83a049d301b4d56e0498e7ac751cb043bf08a5e4982cda97b549d5e6b1aabb05: Status 404 returned error can't find the container with id 83a049d301b4d56e0498e7ac751cb043bf08a5e4982cda97b549d5e6b1aabb05
Sep 09 19:55:02.200004 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.199980    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bab1db_f8ce_4f25_8b38_c48cfae58857.slice/crio-2cbe4d283d9ede2d7ad68f45b1785585e7a719de3dc3c7a4e69372b8cd6fb2c8 WatchSource:0}: Error finding container 2cbe4d283d9ede2d7ad68f45b1785585e7a719de3dc3c7a4e69372b8cd6fb2c8: Status 404 returned error can't find the container with id 2cbe4d283d9ede2d7ad68f45b1785585e7a719de3dc3c7a4e69372b8cd6fb2c8
Sep 09 19:55:02.200857 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.200835    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dfaafcae6539e2d64675470358ee9d.slice/crio-05c1f693366223c14d04a132b9f4db7c4f8fb48c8f626e9b4f3b520fa6737f4e WatchSource:0}: Error finding container 05c1f693366223c14d04a132b9f4db7c4f8fb48c8f626e9b4f3b520fa6737f4e: Status 404 returned error can't find the container with id 05c1f693366223c14d04a132b9f4db7c4f8fb48c8f626e9b4f3b520fa6737f4e
Sep 09 19:55:02.201340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.201308    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7zmp2"
Sep 09 19:55:02.201686 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.201664    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a90e5cd9b3bb35e0b18f98e5a2e76f5.slice/crio-1863fb8592e757102dcbf75374ce176df438a736e31b5ab04f58de3d18c6f74f WatchSource:0}: Error finding container 1863fb8592e757102dcbf75374ce176df438a736e31b5ab04f58de3d18c6f74f: Status 404 returned error can't find the container with id 1863fb8592e757102dcbf75374ce176df438a736e31b5ab04f58de3d18c6f74f
Sep 09 19:55:02.202266 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.202234    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163aa480_c64c_49e6_aa38_b51aabaf7eb8.slice/crio-d08898964658c000265767ef2a405db9092d908b551974c5369f873c88162476 WatchSource:0}: Error finding container d08898964658c000265767ef2a405db9092d908b551974c5369f873c88162476: Status 404 returned error can't find the container with id d08898964658c000265767ef2a405db9092d908b551974c5369f873c88162476
Sep 09 19:55:02.203230 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.203206    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70313ac_da9c_441e_89bb_da79eb54fdb8.slice/crio-462c0aba4ce3d598e3a55c0ce86e0b3f45dc3eec0f0c5cb158a20f80aa331b2e WatchSource:0}: Error finding container 462c0aba4ce3d598e3a55c0ce86e0b3f45dc3eec0f0c5cb158a20f80aa331b2e: Status 404 returned error can't find the container with id 462c0aba4ce3d598e3a55c0ce86e0b3f45dc3eec0f0c5cb158a20f80aa331b2e
Sep 09 19:55:02.205720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.205706    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:55:02.213308 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.213287    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16188a2b_afac_4c06_a671_f786c0937a3e.slice/crio-1afd223868411d01b1f17a0ee9ff9fea95a4646ebbc293fff4fc69dbd08c663b WatchSource:0}: Error finding container 1afd223868411d01b1f17a0ee9ff9fea95a4646ebbc293fff4fc69dbd08c663b: Status 404 returned error can't find the container with id 1afd223868411d01b1f17a0ee9ff9fea95a4646ebbc293fff4fc69dbd08c663b
Sep 09 19:55:02.215996 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.215977    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:02.225922 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.225902    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:02.226170 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.226146    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a10dc16_1c01_4fe5_9fff_1003d62c6ab0.slice/crio-94ecc19d2f8d91797af3641274a1a666cf6bf6ba737e794747e9085433e15df0 WatchSource:0}: Error finding container 94ecc19d2f8d91797af3641274a1a666cf6bf6ba737e794747e9085433e15df0: Status 404 returned error can't find the container with id 94ecc19d2f8d91797af3641274a1a666cf6bf6ba737e794747e9085433e15df0
Sep 09 19:55:02.235673 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.235649    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ee8571_51bc_4fee_a100_6f72cb11057a.slice/crio-92c80e207523fe33f06c6170a6e1b6528db04f4cd747b1fd35be64aaeacc3af8 WatchSource:0}: Error finding container 92c80e207523fe33f06c6170a6e1b6528db04f4cd747b1fd35be64aaeacc3af8: Status 404 returned error can't find the container with id 92c80e207523fe33f06c6170a6e1b6528db04f4cd747b1fd35be64aaeacc3af8
Sep 09 19:55:02.381648 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.381550    2547 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:02.523356 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.523316    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.523385    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.523423    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523477    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523525    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523544    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:02.523567 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523561    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:02.523879 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523576    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:02.523879 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523548    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.523532056 +0000 UTC m=+3.094464749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:02.523879 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523638    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.523620445 +0000 UTC m=+3.094553138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:02.523879 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:02.523655    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.523645123 +0000 UTC m=+3.094577822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:02.729762 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.729524    2547 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:02.792848 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:02.792809    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b50dad_2bbc_42ab_8948_b921b410728a.slice/crio-8d440a6a68f4ca62d767eb66bf922d0ae4add7d81622f6fdc9ee3f6457d8e8b7 WatchSource:0}: Error finding container 8d440a6a68f4ca62d767eb66bf922d0ae4add7d81622f6fdc9ee3f6457d8e8b7: Status 404 returned error can't find the container with id 8d440a6a68f4ca62d767eb66bf922d0ae4add7d81622f6fdc9ee3f6457d8e8b7
Sep 09 19:55:02.856340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.856261    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:02 +0000 UTC" deadline="2028-03-22 14:00:30.528623561 +0000 UTC"
Sep 09 19:55:02.856340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.856338    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13434h5m27.672291018s"
Sep 09 19:55:02.947510 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.947443    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zmp2" event={"ID":"16188a2b-afac-4c06-a671-f786c0937a3e","Type":"ContainerStarted","Data":"1afd223868411d01b1f17a0ee9ff9fea95a4646ebbc293fff4fc69dbd08c663b"}
Sep 09 19:55:02.949509 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.949471    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rx99p" event={"ID":"32bab1db-f8ce-4f25-8b38-c48cfae58857","Type":"ContainerStarted","Data":"2cbe4d283d9ede2d7ad68f45b1785585e7a719de3dc3c7a4e69372b8cd6fb2c8"}
Sep 09 19:55:02.952978 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.952946    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2297" event={"ID":"0c874116-2f48-4afd-92dd-caf462ac9684","Type":"ContainerStarted","Data":"83a049d301b4d56e0498e7ac751cb043bf08a5e4982cda97b549d5e6b1aabb05"}
Sep 09 19:55:02.955909 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.955874    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-plptx" event={"ID":"163aa480-c64c-49e6-aa38-b51aabaf7eb8","Type":"ContainerStarted","Data":"d08898964658c000265767ef2a405db9092d908b551974c5369f873c88162476"}
Sep 09 19:55:02.957859 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.957833    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerStarted","Data":"462c0aba4ce3d598e3a55c0ce86e0b3f45dc3eec0f0c5cb158a20f80aa331b2e"}
Sep 09 19:55:02.960714 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.960688    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal" event={"ID":"0a90e5cd9b3bb35e0b18f98e5a2e76f5","Type":"ContainerStarted","Data":"1863fb8592e757102dcbf75374ce176df438a736e31b5ab04f58de3d18c6f74f"}
Sep 09 19:55:02.962899 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.962873    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8qhwg" event={"ID":"77b50dad-2bbc-42ab-8948-b921b410728a","Type":"ContainerStarted","Data":"8d440a6a68f4ca62d767eb66bf922d0ae4add7d81622f6fdc9ee3f6457d8e8b7"}
Sep 09 19:55:02.964382 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.964356    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal" event={"ID":"e3dfaafcae6539e2d64675470358ee9d","Type":"ContainerStarted","Data":"05c1f693366223c14d04a132b9f4db7c4f8fb48c8f626e9b4f3b520fa6737f4e"}
Sep 09 19:55:02.966768 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.966746    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"92c80e207523fe33f06c6170a6e1b6528db04f4cd747b1fd35be64aaeacc3af8"}
Sep 09 19:55:02.968252 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:02.968227    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rsnjr" event={"ID":"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0","Type":"ContainerStarted","Data":"94ecc19d2f8d91797af3641274a1a666cf6bf6ba737e794747e9085433e15df0"}
Sep 09 19:55:03.531732 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.531626    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:03.531732 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.531699    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:03.531732 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.531729    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:03.532010 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.531910    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:03.532010 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.531937    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:03.532010 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.531951    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:03.532156 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.532021    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.531999402 +0000 UTC m=+5.102932112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:03.532528 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.532501    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:03.532650 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.532556    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.532542635 +0000 UTC m=+5.103475330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:03.532650 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.532618    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:03.532770 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.532655    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.53264269 +0000 UTC m=+5.103575424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:03.856689 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.856593    2547 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:50:02 +0000 UTC" deadline="2028-05-24 01:23:58.428582004 +0000 UTC"
Sep 09 19:55:03.856689 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.856638    2547 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14933h28m54.571948236s"
Sep 09 19:55:03.941937 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.941891    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:03.942122 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.942039    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:03.943455 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.942544    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:03.943455 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.942665    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:03.943455 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.942934    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:03.943455 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:03.943034    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:03.985794 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:03.985756    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k" event={"ID":"35796d71-fc30-4fe5-8b84-6355b621efff","Type":"ContainerStarted","Data":"cca3bbe823b599409ea72466e6fddd1e6f65fd56dce39d1060b2b3f9560e53e0"}
Sep 09 19:55:04.211465 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:04.211376    2547 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:55:05.547504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.547225    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:05.548007 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.547575    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:05.548007 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.547629    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:05.548007 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.547760    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:05.548007 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.547827    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:09.547807171 +0000 UTC m=+9.118739872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:05.548314 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548275    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:05.548314 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548298    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:05.548314 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548312    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:05.548549 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548360    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:09.548344251 +0000 UTC m=+9.119276961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:05.548549 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548437    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:05.548549 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.548473    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:09.54846112 +0000 UTC m=+9.119393828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:05.941145 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.941060    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:05.941321 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.941060    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:05.941321 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.941207    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:05.941462 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.941313    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:05.941462 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:05.941081    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:05.941577 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:05.941465    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:07.941735 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:07.941691    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:07.942240 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:07.941833    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:07.942317 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:07.942299    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:07.942451 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:07.942430    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:07.942510 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:07.942493    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:07.942587 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:07.942569    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:09.587041 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.586998    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.587079    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.587115    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587182    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587263    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.587240545 +0000 UTC m=+17.158173254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587296    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587316    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587348    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587401    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.58738413 +0000 UTC m=+17.158316829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587472    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:09.587691 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.587507    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:17.587495346 +0000 UTC m=+17.158428045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:09.941724 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.941644    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:09.941906 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.941791    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:09.942217 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.942198    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:09.942322 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.942299    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:09.942405 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:09.942384    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:09.942492 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:09.942475    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:11.941708 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:11.941662    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:11.942179 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:11.941717    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:11.942179 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:11.941792    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:11.942179 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:11.941808    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:11.942179 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:11.941903    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:11.942179 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:11.941974    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:13.941276 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:13.941246    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:13.941739 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:13.941437    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:13.941739 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:13.941446    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:13.941739 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:13.941474    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:13.941739 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:13.941554    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:13.941739 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:13.941641    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:15.941866 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:15.941827    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:15.941866 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:15.941872    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:15.942375 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:15.941960    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:15.942375 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:15.941988    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:15.942375 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:15.942085    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:15.942375 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:15.942171    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:17.647874 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.647834    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:17.647874 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.647884    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.647902    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648004    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648020    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648024    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648042    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648053    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648097    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.648078568 +0000 UTC m=+33.219011260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648113    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.648106113 +0000 UTC m=+33.219038805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:17.648425 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.648124    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.648118409 +0000 UTC m=+33.219051100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:17.941502 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.941410    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:17.941502 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.941443    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:17.941502 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:17.941454    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:17.941779 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.941535    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:17.941779 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.941598    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:17.941779 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:17.941690    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:19.941248 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:19.941215    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:19.941248 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:19.941237    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:19.941869 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:19.941242    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:19.941869 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:19.941358    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:19.941869 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:19.941410    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:19.941869 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:19.941498    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:21.941675 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:21.941639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:21.942071 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:21.941639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:21.942071 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:21.941748    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:21.942071 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:21.941808    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:21.942071 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:21.941639    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:21.942071 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:21.941879    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:23.025646 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.025403    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="132c031ac05b9554b3e1708747180ac937a3ebed6852c4726030caeea5329d82" exitCode=0
Sep 09 19:55:23.026482 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.025493    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"132c031ac05b9554b3e1708747180ac937a3ebed6852c4726030caeea5329d82"}
Sep 09 19:55:23.027428 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.027397    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal" event={"ID":"0a90e5cd9b3bb35e0b18f98e5a2e76f5","Type":"ContainerStarted","Data":"46a8c7e999553135a770c98512e1d796ded0429d26bc34bfa7cc8a6325ebd74e"}
Sep 09 19:55:23.029020 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.028997    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8qhwg" event={"ID":"77b50dad-2bbc-42ab-8948-b921b410728a","Type":"ContainerStarted","Data":"3dad2535441cb87f281931194a7656c481e5a41a4b5847864c7e92385af88681"}
Sep 09 19:55:23.030507 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.030486    2547 generic.go:356] "Generic (PLEG): container finished" podID="e3dfaafcae6539e2d64675470358ee9d" containerID="c92eb6e8201d7ae8db1dda6e772146285ec0e05e060eb22889cf2c6ab5c58d5a" exitCode=0
Sep 09 19:55:23.030595 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.030556    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal" event={"ID":"e3dfaafcae6539e2d64675470358ee9d","Type":"ContainerDied","Data":"c92eb6e8201d7ae8db1dda6e772146285ec0e05e060eb22889cf2c6ab5c58d5a"}
Sep 09 19:55:23.032221 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.032195    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k" event={"ID":"35796d71-fc30-4fe5-8b84-6355b621efff","Type":"ContainerStarted","Data":"e303c123bbe0f7e25226cd467a5631d3b506b5e8caeaaead4b47f56eb4b6b02c"}
Sep 09 19:55:23.035167 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035147    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"c3f7b5f8878d7aac0a7b28cb98c570a9d18547e6ef2719d849c1a5842e3f6dae"}
Sep 09 19:55:23.035258 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035242    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"b4fe7939cca7e4bffb5dabbbcf3b3fc3b4645c549b40d41e63fdda8115fb9a6b"}
Sep 09 19:55:23.035342 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"cfa9fdcc4e8213ef6fe73ffdb11f02e16c4493ae58cf2fab68ce563f9ca4d9d9"}
Sep 09 19:55:23.035401 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035373    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"dccdb043f565ddc37dae3b1248880c2d443e8945ba86594689da9c789a0572e5"}
Sep 09 19:55:23.035457 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035416    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"bb8208ffb0fac2cb2b7c053a470eee8704065deea3de8cdcb50a280f8b39e5ae"}
Sep 09 19:55:23.035504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.035457    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"fe4ab4a79fc5b0d8d2d7eab01e1306c001099f1afded602444481c5dfddd03a4"}
Sep 09 19:55:23.036780 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.036760    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rsnjr" event={"ID":"5a10dc16-1c01-4fe5-9fff-1003d62c6ab0","Type":"ContainerStarted","Data":"679b8f5647bdec93bf840bccbc98cae622198e38a56e28453ffda4026969ee12"}
Sep 09 19:55:23.038353 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.038324    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rx99p" event={"ID":"32bab1db-f8ce-4f25-8b38-c48cfae58857","Type":"ContainerStarted","Data":"985ed657f70b1e4757f046caeeb06722ecb9ddccdac2a85d93b1713e3b731558"}
Sep 09 19:55:23.039792 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.039772    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2297" event={"ID":"0c874116-2f48-4afd-92dd-caf462ac9684","Type":"ContainerStarted","Data":"2566cb8937734fb30074e52ac011b90b472910eec26c6149cf30b80f39508276"}
Sep 09 19:55:23.041694 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.041675    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-plptx" event={"ID":"163aa480-c64c-49e6-aa38-b51aabaf7eb8","Type":"ContainerStarted","Data":"80518599e243d5505e6fed7778ed62091652a8b4c2ea94a16d8780e1d808b52c"}
Sep 09 19:55:23.078868 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.078825    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-11.ec2.internal" podStartSLOduration=21.078809344 podStartE2EDuration="21.078809344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.061742444 +0000 UTC m=+22.632675158" watchObservedRunningTime="2026-09-09 19:55:23.078809344 +0000 UTC m=+22.649742059"
Sep 09 19:55:23.079070 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.079047    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2297" podStartSLOduration=6.745240224 podStartE2EDuration="22.07903905s" totalImagesPullingTime="15.333798826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.078603198 +0000 UTC m=+22.649535913" watchObservedRunningTime="2026-09-09 19:55:23.07903905 +0000 UTC m=+22.649971763"
Sep 09 19:55:23.109629 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.109577    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-plptx" podStartSLOduration=2.094792983 podStartE2EDuration="22.109554873s" totalImagesPullingTime="20.01476189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.094533154 +0000 UTC m=+22.665465884" watchObservedRunningTime="2026-09-09 19:55:23.109554873 +0000 UTC m=+22.680487589"
Sep 09 19:55:23.110059 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.110030    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rsnjr" podStartSLOduration=6.798124331 podStartE2EDuration="22.110018528s" totalImagesPullingTime="15.311894197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.109014554 +0000 UTC m=+22.679947268" watchObservedRunningTime="2026-09-09 19:55:23.110018528 +0000 UTC m=+22.680951243"
Sep 09 19:55:23.141778 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.141737    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8qhwg" podStartSLOduration=2.720008639 podStartE2EDuration="22.141721739s" totalImagesPullingTime="19.4217131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.14146858 +0000 UTC m=+22.712401321" watchObservedRunningTime="2026-09-09 19:55:23.141721739 +0000 UTC m=+22.712654450"
Sep 09 19:55:23.162267 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.161593    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rx99p" podStartSLOduration=1.9948913209999999 podStartE2EDuration="22.161573374s" totalImagesPullingTime="20.166682053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.161240404 +0000 UTC m=+22.732173119" watchObservedRunningTime="2026-09-09 19:55:23.161573374 +0000 UTC m=+22.732506089"
Sep 09 19:55:23.941260 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.941182    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:23.941435 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.941182    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:23.941435 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:23.941328    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:23.941435 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:23.941418    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:23.941587 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:23.941182    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:23.941587 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:23.941512    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:24.046363 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.046323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal" event={"ID":"e3dfaafcae6539e2d64675470358ee9d","Type":"ContainerStarted","Data":"22e764469730cde6eaeb3fd679d26bf645d45cf34e1f1c629e4f79fd445c9269"}
Sep 09 19:55:24.048807 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.048777    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7zmp2" event={"ID":"16188a2b-afac-4c06-a671-f786c0937a3e","Type":"ContainerStarted","Data":"be5b25f29d7149aa807e67b03d2183d4e7f5c7e44ffd3ed0b3436936a2988551"}
Sep 09 19:55:24.062091 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.062037    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-11.ec2.internal" podStartSLOduration=22.062020249 podStartE2EDuration="22.062020249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:24.061943763 +0000 UTC m=+23.632876503" watchObservedRunningTime="2026-09-09 19:55:24.062020249 +0000 UTC m=+23.632952963"
Sep 09 19:55:24.075983 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.075941    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7zmp2" podStartSLOduration=3.070564921 podStartE2EDuration="23.075928099s" totalImagesPullingTime="20.005363178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:24.075137034 +0000 UTC m=+23.646069764" watchObservedRunningTime="2026-09-09 19:55:24.075928099 +0000 UTC m=+23.646860815"
Sep 09 19:55:24.250545 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.250509    2547 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:24.872658 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.872429    2547 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:24.250529452Z","UUID":"16c1c753-fcb1-4489-afc4-b96767695db8","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:24.874643 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.874621    2547 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:24.874796 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:24.874658    2547 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:25.053156 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:25.053055    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k" event={"ID":"35796d71-fc30-4fe5-8b84-6355b621efff","Type":"ContainerStarted","Data":"e6823c3e6c53223e4258a2d0bc882260b2cd9c77e1a68d452b0f7810cd7143ab"}
Sep 09 19:55:25.941748 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:25.941504    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:25.941929 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:25.941555    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:25.941929 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:25.941562    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:25.941929 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:25.941846    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:25.941929 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:25.941913    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:25.942086 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:25.942003    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:26.058942 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:26.058895    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"d828a00405869faf2f6b8bdb8a0d027560dad43586a7434c389ba274ed73411b"}
Sep 09 19:55:26.092192 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:26.092160    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:26.092862 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:26.092839    2547 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:27.061280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:27.061247    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:27.061782 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:27.061766    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rsnjr"
Sep 09 19:55:27.941883 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:27.941690    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:27.942029 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:27.941713    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:27.942029 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:27.941978    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:27.942102 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:27.941720    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:27.942102 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:27.942023    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:27.942193 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:27.942129    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:28.065222 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.065185    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k" event={"ID":"35796d71-fc30-4fe5-8b84-6355b621efff","Type":"ContainerStarted","Data":"af01561ef743d0207c36864a28a4e9ebcaf61464c62d66c0f4ab90b4f58c96cb"}
Sep 09 19:55:28.069004 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.068955    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" event={"ID":"29ee8571-51bc-4fee-a100-6f72cb11057a","Type":"ContainerStarted","Data":"1f9c90c9ec5d0d3e5ecc594db71818f632090e66bd37d761ca4ccd42667eb2ca"}
Sep 09 19:55:28.069223 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.069206    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:28.069358 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.069278    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:28.069358 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.069348    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:28.071193 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.071170    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="31f523f07c1c08d55d74a4d896a4218bcc86d7c7bd392ce20aca3f5037544587" exitCode=0
Sep 09 19:55:28.071336 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.071274    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"31f523f07c1c08d55d74a4d896a4218bcc86d7c7bd392ce20aca3f5037544587"}
Sep 09 19:55:28.085054 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.085032    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:28.085136 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.085126    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:55:28.085924 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.085890    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gb82k" podStartSLOduration=3.6749334769999997 podStartE2EDuration="27.08587775s" totalImagesPullingTime="23.410944273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:28.085834589 +0000 UTC m=+27.656767304" watchObservedRunningTime="2026-09-09 19:55:28.08587775 +0000 UTC m=+27.656810465"
Sep 09 19:55:28.137436 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:28.137385    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm" podStartSLOduration=7.153547506 podStartE2EDuration="27.137368957s" totalImagesPullingTime="19.983821451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:28.136753098 +0000 UTC m=+27.707685811" watchObservedRunningTime="2026-09-09 19:55:28.137368957 +0000 UTC m=+27.708301672"
Sep 09 19:55:29.076125 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.076091    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="4b5cb01c63a4731963a4c9f36265652c0dd2ee72c84f7c939a54937bb629a6d9" exitCode=0
Sep 09 19:55:29.076710 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.076182    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"4b5cb01c63a4731963a4c9f36265652c0dd2ee72c84f7c939a54937bb629a6d9"}
Sep 09 19:55:29.405782 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.405750    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfnf2"]
Sep 09 19:55:29.405963 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.405895    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:29.406033 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:29.406001    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:29.408482 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.408451    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q9n5s"]
Sep 09 19:55:29.408593 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.408544    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:29.408671 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:29.408646    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:29.418284 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.418258    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm8vh"]
Sep 09 19:55:29.418412 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:29.418377    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:29.418495 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:29.418474    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:30.081439 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:30.081345    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="7944637356d51bfdac195b5d6e7b08f706c3b9800dcf5c57d853c84602a182ad" exitCode=0
Sep 09 19:55:30.081439 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:30.081419    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"7944637356d51bfdac195b5d6e7b08f706c3b9800dcf5c57d853c84602a182ad"}
Sep 09 19:55:30.942642 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:30.942427    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:30.942816 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:30.942483    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:30.942816 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:30.942719    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:30.942816 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:30.942794    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:30.942816 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:30.942507    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:30.943034 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:30.942875    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:32.941969 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:32.941929    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:32.941969 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:32.941967    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:32.942501 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:32.942077    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:32.942501 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:32.942158    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:32.942501 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:32.942178    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:32.942501 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:32.942282    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:33.665475 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:33.665437    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:33.665503    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:33.665539    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665614    2547 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665646    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665659    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665676    2547 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:33.665683 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665687    2547 projected.go:196] Error preparing data for projected volume kube-api-access-vtghr for pod openshift-network-diagnostics/network-check-target-cfnf2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:33.665998 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665691    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret podName:99c7cf30-f214-4557-87ce-05025c7000eb nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.665669121 +0000 UTC m=+65.236601818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret") pod "global-pull-secret-syncer-sm8vh" (UID: "99c7cf30-f214-4557-87ce-05025c7000eb") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:33.665998 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665711    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.665700803 +0000 UTC m=+65.236633502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:33.665998 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:33.665730    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr podName:f993fcdf-c198-419d-a123-b1daf4c877e9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.665724553 +0000 UTC m=+65.236657245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtghr" (UniqueName: "kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr") pod "network-check-target-cfnf2" (UID: "f993fcdf-c198-419d-a123-b1daf4c877e9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:34.941095 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:34.941058    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:34.941095 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:34.941077    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:34.941095 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:34.941067    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:34.941767 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:34.941206    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm8vh" podUID="99c7cf30-f214-4557-87ce-05025c7000eb"
Sep 09 19:55:34.941767 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:34.941279    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:55:34.941767 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:34.941326    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cfnf2" podUID="f993fcdf-c198-419d-a123-b1daf4c877e9"
Sep 09 19:55:35.096404 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:35.096351    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="1ea659975780ae0a80e57c298022ae338e75329a078e41af97a74ca41b7bfaee" exitCode=0
Sep 09 19:55:35.096570 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:35.096434    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"1ea659975780ae0a80e57c298022ae338e75329a078e41af97a74ca41b7bfaee"}
Sep 09 19:55:36.102454 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.102368    2547 generic.go:356] "Generic (PLEG): container finished" podID="a70313ac-da9c-441e-89bb-da79eb54fdb8" containerID="9437bf249a818ed094a9a09723e7b25c96be8929bb22d1e4d57ad6eb9047c6f8" exitCode=0
Sep 09 19:55:36.102454 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.102413    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerDied","Data":"9437bf249a818ed094a9a09723e7b25c96be8929bb22d1e4d57ad6eb9047c6f8"}
Sep 09 19:55:36.207237 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.207211    2547 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-11.ec2.internal" event="NodeReady"
Sep 09 19:55:36.207415 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.207385    2547 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:36.252210 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.252176    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zwrn5"]
Sep 09 19:55:36.265390 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.265366    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xhjbb"]
Sep 09 19:55:36.265500 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.265484    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.267879 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.267671    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:36.267879 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.267759    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:36.267879 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.267786    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hd98l\""
Sep 09 19:55:36.268633 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.268614    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:36.275667 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.275480    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xhjbb"]
Sep 09 19:55:36.275758 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.275694    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwrn5"]
Sep 09 19:55:36.275758 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.275588    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.277734 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.277716    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:36.278071 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.278050    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:36.278372 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.278346    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4z7jm\""
Sep 09 19:55:36.386109 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386079    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01869d12-9e3b-4439-84bb-173a4f515694-config-volume\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.386280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386122    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01869d12-9e3b-4439-84bb-173a4f515694-tmp-dir\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.386280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386168    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-cert\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.386280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386206    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw6hr\" (UniqueName: \"kubernetes.io/projected/01869d12-9e3b-4439-84bb-173a4f515694-kube-api-access-bw6hr\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.386280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386235    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.386280 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.386266    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttcc\" (UniqueName: \"kubernetes.io/projected/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-kube-api-access-qttcc\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.487376 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487334    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.487548 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487396    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttcc\" (UniqueName: \"kubernetes.io/projected/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-kube-api-access-qttcc\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.487548 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487428    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01869d12-9e3b-4439-84bb-173a4f515694-config-volume\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.487548 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487450    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01869d12-9e3b-4439-84bb-173a4f515694-tmp-dir\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.487548 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:36.487488    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:36.487738 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:36.487569    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:36.987547047 +0000 UTC m=+36.558479739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:36.487738 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487487    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-cert\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.487738 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.487605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw6hr\" (UniqueName: \"kubernetes.io/projected/01869d12-9e3b-4439-84bb-173a4f515694-kube-api-access-bw6hr\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.488041 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.488021    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01869d12-9e3b-4439-84bb-173a4f515694-tmp-dir\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.488345 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.488323    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01869d12-9e3b-4439-84bb-173a4f515694-config-volume\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.491587 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.491570    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-cert\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.495235 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.495208    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttcc\" (UniqueName: \"kubernetes.io/projected/0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c-kube-api-access-qttcc\") pod \"ingress-canary-zwrn5\" (UID: \"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c\") " pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.495486 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.495467    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw6hr\" (UniqueName: \"kubernetes.io/projected/01869d12-9e3b-4439-84bb-173a4f515694-kube-api-access-bw6hr\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.575544 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.575504    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zwrn5"
Sep 09 19:55:36.904279 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.904245    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zwrn5"]
Sep 09 19:55:36.908090 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:55:36.908060    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b82ff8a_392c_4e58_bbc4_f0ec10f08a2c.slice/crio-7979aa4cad209300dc49993c19e491d306d7c0209ee3cae5cc0a794730e5de2b WatchSource:0}: Error finding container 7979aa4cad209300dc49993c19e491d306d7c0209ee3cae5cc0a794730e5de2b: Status 404 returned error can't find the container with id 7979aa4cad209300dc49993c19e491d306d7c0209ee3cae5cc0a794730e5de2b
Sep 09 19:55:36.941451 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.941370    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:55:36.941451 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.941372    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:55:36.941451 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.941372    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:55:36.943808 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.943786    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:36.944430 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.944410    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:36.944533 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.944434    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bbhmn\""
Sep 09 19:55:36.944533 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.944417    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cpmqx\""
Sep 09 19:55:36.944708 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.944693    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:36.944776 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.944736    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:36.990942 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:36.990920    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:36.991071 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:36.991058    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:36.991128 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:36.991118    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.991100317 +0000 UTC m=+37.562033015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:37.106116 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:37.106078    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwrn5" event={"ID":"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c","Type":"ContainerStarted","Data":"7979aa4cad209300dc49993c19e491d306d7c0209ee3cae5cc0a794730e5de2b"}
Sep 09 19:55:37.109309 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:37.109258    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-svpxv" event={"ID":"a70313ac-da9c-441e-89bb-da79eb54fdb8","Type":"ContainerStarted","Data":"97a1bd1e3e2a9d1ffb2eeaf34cd711ab3363c3c8e4504ce394af8e8194c66085"}
Sep 09 19:55:37.129343 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:37.129274    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-svpxv" podStartSLOduration=6.258872587 podStartE2EDuration="36.129261823s" totalImagesPullingTime="29.870389236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.128430272 +0000 UTC m=+36.699362986" watchObservedRunningTime="2026-09-09 19:55:37.129261823 +0000 UTC m=+36.700194537"
Sep 09 19:55:37.997343 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:37.997305    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:37.997555 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:37.997474    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:37.997625 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:37.997563    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.997538724 +0000 UTC m=+39.568471422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:40.012920 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:40.012879    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:40.013455 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:40.013007    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:40.013455 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:40.013062    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:44.013047087 +0000 UTC m=+43.583979780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:40.118118 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:40.118083    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zwrn5" event={"ID":"0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c","Type":"ContainerStarted","Data":"b4c5ddde102f1819c8a4912ce7362de97e4a5890ee9a827f79a084d505ddfd29"}
Sep 09 19:55:40.136213 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:40.136148    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zwrn5" podStartSLOduration=1.7875433090000001 podStartE2EDuration="4.136132786s" totalImagesPullingTime="2.348589477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:40.135554138 +0000 UTC m=+39.706486852" watchObservedRunningTime="2026-09-09 19:55:40.136132786 +0000 UTC m=+39.707065499"
Sep 09 19:55:44.039205 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:44.039161    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:44.039649 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:44.039346    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:44.039649 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:44.039407    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:52.039390198 +0000 UTC m=+51.610322895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:52.098859 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:55:52.098820    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:55:52.099365 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:52.098964    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:52.099365 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:55:52.099024    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:08.0990069 +0000 UTC m=+67.669939599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:00.091741 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:00.091712    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26bsm"
Sep 09 19:56:05.693409 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.693363    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:56:05.693409 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.693414    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:56:05.693937 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.693465    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:56:05.695813 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.695790    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:05.695866 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.695816    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:05.695905 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.695867    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:56:05.704503 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:05.704484    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:56:05.704575 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:05.704544    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:57:09.704526297 +0000 UTC m=+129.275458989 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : secret "metrics-daemon-secret" not found
Sep 09 19:56:05.705986 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.705970    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:56:05.707517 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.707496    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99c7cf30-f214-4557-87ce-05025c7000eb-original-pull-secret\") pod \"global-pull-secret-syncer-sm8vh\" (UID: \"99c7cf30-f214-4557-87ce-05025c7000eb\") " pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:56:05.717698 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.717671    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtghr\" (UniqueName: \"kubernetes.io/projected/f993fcdf-c198-419d-a123-b1daf4c877e9-kube-api-access-vtghr\") pod \"network-check-target-cfnf2\" (UID: \"f993fcdf-c198-419d-a123-b1daf4c877e9\") " pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:56:05.752584 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.752555    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm8vh"
Sep 09 19:56:05.759512 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.759491    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bbhmn\""
Sep 09 19:56:05.767867 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:05.767849    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:56:06.048720 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:06.048639    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm8vh"]
Sep 09 19:56:06.052279 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:56:06.052249    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c7cf30_f214_4557_87ce_05025c7000eb.slice/crio-5bfb33e76e738171ed70960130511cf319f82c45e0aa83b51c903ecf0117a478 WatchSource:0}: Error finding container 5bfb33e76e738171ed70960130511cf319f82c45e0aa83b51c903ecf0117a478: Status 404 returned error can't find the container with id 5bfb33e76e738171ed70960130511cf319f82c45e0aa83b51c903ecf0117a478
Sep 09 19:56:06.069146 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:06.069122    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cfnf2"]
Sep 09 19:56:06.072320 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:56:06.072274    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993fcdf_c198_419d_a123_b1daf4c877e9.slice/crio-6493c3b07cae883216bb1fa56e4eb8f8121978abecbdbb926089667d9c53a878 WatchSource:0}: Error finding container 6493c3b07cae883216bb1fa56e4eb8f8121978abecbdbb926089667d9c53a878: Status 404 returned error can't find the container with id 6493c3b07cae883216bb1fa56e4eb8f8121978abecbdbb926089667d9c53a878
Sep 09 19:56:06.177188 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:06.177153    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfnf2" event={"ID":"f993fcdf-c198-419d-a123-b1daf4c877e9","Type":"ContainerStarted","Data":"6493c3b07cae883216bb1fa56e4eb8f8121978abecbdbb926089667d9c53a878"}
Sep 09 19:56:06.178348 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:06.178323    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm8vh" event={"ID":"99c7cf30-f214-4557-87ce-05025c7000eb","Type":"ContainerStarted","Data":"5bfb33e76e738171ed70960130511cf319f82c45e0aa83b51c903ecf0117a478"}
Sep 09 19:56:08.109357 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:08.109323    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:56:08.109809 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:08.109464    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:08.109809 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:08.109527    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:40.109509531 +0000 UTC m=+99.680442230 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:11.194331 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:11.194218    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cfnf2" event={"ID":"f993fcdf-c198-419d-a123-b1daf4c877e9","Type":"ContainerStarted","Data":"4f2cc12903ef0d70fcba04ba06e76c510c2d427dc98907fbdfc8ec412e8c914d"}
Sep 09 19:56:11.194783 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:11.194342    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:56:11.195783 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:11.195761    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm8vh" event={"ID":"99c7cf30-f214-4557-87ce-05025c7000eb","Type":"ContainerStarted","Data":"6de4dd46e147976145194b9c9b48e44358d1e8002e94531b6fd511359c81135d"}
Sep 09 19:56:11.209187 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:11.209133    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cfnf2" podStartSLOduration=65.350594161 podStartE2EDuration="1m10.209120751s" totalImagesPullingTime="4.85852659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:11.208802157 +0000 UTC m=+70.779734872" watchObservedRunningTime="2026-09-09 19:56:11.209120751 +0000 UTC m=+70.780053533"
Sep 09 19:56:11.223371 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:11.223328    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sm8vh" podStartSLOduration=64.341655071 podStartE2EDuration="1m9.223314758s" totalImagesPullingTime="4.881659687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:11.222858285 +0000 UTC m=+70.793790997" watchObservedRunningTime="2026-09-09 19:56:11.223314758 +0000 UTC m=+70.794247472"
Sep 09 19:56:40.114012 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.113963    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:56:40.114488 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:40.114109    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:40.114488 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:56:40.114182    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:57:44.11416562 +0000 UTC m=+163.685098312 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:40.784267 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.784237    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"]
Sep 09 19:56:40.788269 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.788246    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.791365 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.791342    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 19:56:40.791365 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.791356    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:40.791365 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.791363    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:40.791577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.791342    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:40.791577 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.791346    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8xr46\""
Sep 09 19:56:40.795672 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.795652    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"]
Sep 09 19:56:40.798275 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.798262    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.800340 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.800320    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"]
Sep 09 19:56:40.800700 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.800683    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 19:56:40.800806 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.800790    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 19:56:40.800866 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.800810    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 19:56:40.801017 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.801003    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 19:56:40.815346 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.815324    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"]
Sep 09 19:56:40.817757 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.817736    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c5ba203-2f8c-4273-a9d8-ee8828878923-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.817849 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.817770    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtt7\" (UniqueName: \"kubernetes.io/projected/9c5ba203-2f8c-4273-a9d8-ee8828878923-kube-api-access-dhtt7\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.918851 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.918818    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.918851 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.918852    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.919042 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.918877    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.919042 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.918915    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c5ba203-2f8c-4273-a9d8-ee8828878923-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.919107 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.919066    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtt7\" (UniqueName: \"kubernetes.io/projected/9c5ba203-2f8c-4273-a9d8-ee8828878923-kube-api-access-dhtt7\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.919107 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.919093    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knzzg\" (UniqueName: \"kubernetes.io/projected/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-kube-api-access-knzzg\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.919169 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.919129    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.919169 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.919152    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:40.921415 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.921396    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c5ba203-2f8c-4273-a9d8-ee8828878923-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:40.927715 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:40.927690    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtt7\" (UniqueName: \"kubernetes.io/projected/9c5ba203-2f8c-4273-a9d8-ee8828878923-kube-api-access-dhtt7\") pod \"managed-serviceaccount-addon-agent-8565bfbcd4-s72wm\" (UID: \"9c5ba203-2f8c-4273-a9d8-ee8828878923\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:41.019925 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.019893    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.019925 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.019923    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.020089 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.019942    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.020089 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.019970    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knzzg\" (UniqueName: \"kubernetes.io/projected/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-kube-api-access-knzzg\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.020250 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.020096    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.020250 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.020139    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.020873 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.020849    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.022388 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.022362    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-ca\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.022492 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.022458    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.022538 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.022512    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-hub\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.022593 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.022574    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.029001 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.028978    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knzzg\" (UniqueName: \"kubernetes.io/projected/ee7541f0-bee8-4b77-98d3-1c83898f2fc1-kube-api-access-knzzg\") pod \"cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q\" (UID: \"ee7541f0-bee8-4b77-98d3-1c83898f2fc1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.108613 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.108542    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"
Sep 09 19:56:41.115392 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.115374    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 19:56:41.382902 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.382869    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm"]
Sep 09 19:56:41.385985 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:56:41.385956    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c5ba203_2f8c_4273_a9d8_ee8828878923.slice/crio-e7339b832841a1774dd5eaddd97380f0d8702a408f3d73c1b0b610df86c9ea35 WatchSource:0}: Error finding container e7339b832841a1774dd5eaddd97380f0d8702a408f3d73c1b0b610df86c9ea35: Status 404 returned error can't find the container with id e7339b832841a1774dd5eaddd97380f0d8702a408f3d73c1b0b610df86c9ea35
Sep 09 19:56:41.391896 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:41.391875    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"]
Sep 09 19:56:41.397053 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:56:41.397029    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7541f0_bee8_4b77_98d3_1c83898f2fc1.slice/crio-b69c80adc50c5ef91d271db687906994fd75f281b90dc964c3485c75dc6c7df5 WatchSource:0}: Error finding container b69c80adc50c5ef91d271db687906994fd75f281b90dc964c3485c75dc6c7df5: Status 404 returned error can't find the container with id b69c80adc50c5ef91d271db687906994fd75f281b90dc964c3485c75dc6c7df5
Sep 09 19:56:42.200533 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:42.200501    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cfnf2"
Sep 09 19:56:42.270504 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:42.270463    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerStarted","Data":"b69c80adc50c5ef91d271db687906994fd75f281b90dc964c3485c75dc6c7df5"}
Sep 09 19:56:42.271864 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:42.271837    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm" event={"ID":"9c5ba203-2f8c-4273-a9d8-ee8828878923","Type":"ContainerStarted","Data":"e7339b832841a1774dd5eaddd97380f0d8702a408f3d73c1b0b610df86c9ea35"}
Sep 09 19:56:47.291557 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.291524    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm" event={"ID":"9c5ba203-2f8c-4273-a9d8-ee8828878923","Type":"ContainerStarted","Data":"9f4c7b74b9cee9cf366c985b77d56677f323960d1f33ee27f4e68b8f3f34e016"}
Sep 09 19:56:47.293613 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.293588    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerStarted","Data":"6c502d1a3175321d60110d840e137069aa64a765a57f72d4eecdd4ec2e9679e2"}
Sep 09 19:56:47.293723 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.293628    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerStarted","Data":"d1fdd164160417c83181856cdb6e487f8dd6fad45e9ac2d7e8c431642ab12f66"}
Sep 09 19:56:47.293723 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.293640    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerStarted","Data":"a5da9d9f36fdb6d75598f695fba7588c90aaf0472cee7ae73911196bdb4ab04e"}
Sep 09 19:56:47.308229 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.308185    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm" podStartSLOduration=2.182580225 podStartE2EDuration="7.308171184s" totalImagesPullingTime="5.125590959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:47.307546754 +0000 UTC m=+106.878479469" watchObservedRunningTime="2026-09-09 19:56:47.308171184 +0000 UTC m=+106.879103898"
Sep 09 19:56:47.327037 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:56:47.326966    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" podStartSLOduration=2.197331467 podStartE2EDuration="7.326923193s" totalImagesPullingTime="5.129591726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:47.326498189 +0000 UTC m=+106.897430903" watchObservedRunningTime="2026-09-09 19:56:47.326923193 +0000 UTC m=+106.897855907"
Sep 09 19:57:09.722314 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:57:09.722256    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:57:09.722806 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:09.722417    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:57:09.722806 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:09.722498    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 19:59:11.722480196 +0000 UTC m=+251.293412888 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : secret "metrics-daemon-secret" not found
Sep 09 19:57:39.284159 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:39.284043    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xhjbb" podUID="01869d12-9e3b-4439-84bb-173a4f515694"
Sep 09 19:57:39.443549 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:57:39.443519    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:57:39.962663 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:39.962622    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:57:44.158386 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:57:44.158340    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:57:44.158853 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:44.158492    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:57:44.158853 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:57:44.158571    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:46.158554643 +0000 UTC m=+285.729487336 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:57:51.941784 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:57:51.941743    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:58:47.638815 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:58:47.638781    2547 generic.go:356] "Generic (PLEG): container finished" podID="9c5ba203-2f8c-4273-a9d8-ee8828878923" containerID="9f4c7b74b9cee9cf366c985b77d56677f323960d1f33ee27f4e68b8f3f34e016" exitCode=255
Sep 09 19:58:47.638815 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:58:47.638818    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm" event={"ID":"9c5ba203-2f8c-4273-a9d8-ee8828878923","Type":"ContainerDied","Data":"9f4c7b74b9cee9cf366c985b77d56677f323960d1f33ee27f4e68b8f3f34e016"}
Sep 09 19:58:47.639263 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:58:47.639106    2547 scope.go:117] "RemoveContainer" containerID="9f4c7b74b9cee9cf366c985b77d56677f323960d1f33ee27f4e68b8f3f34e016"
Sep 09 19:58:48.643460 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:58:48.643422    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8565bfbcd4-s72wm" event={"ID":"9c5ba203-2f8c-4273-a9d8-ee8828878923","Type":"ContainerStarted","Data":"468856b9414895f0d79248b58c7f87f472730045df697dc803f0cc56fb262874"}
Sep 09 19:59:11.743714 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:11.743673    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 19:59:11.744173 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:11.743824    2547 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:59:11.744173 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:11.743896    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs podName:90d331aa-5052-4a95-91cc-b82896d0d9df nodeName:}" failed. No retries permitted until 2026-09-09 20:01:13.743879595 +0000 UTC m=+373.314812287 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs") pod "network-metrics-daemon-q9n5s" (UID: "90d331aa-5052-4a95-91cc-b82896d0d9df") : secret "metrics-daemon-secret" not found
Sep 09 19:59:38.544017 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.543932    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8bclv"]
Sep 09 19:59:38.546908 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.546890    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.548806 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.548783    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:59:38.548907 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.548790    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xxlzl\""
Sep 09 19:59:38.549548 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.549532    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:59:38.549625 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.549547    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:59:38.549625 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.549538    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:59:38.555942 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.555922    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bclv"]
Sep 09 19:59:38.721713 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.721673    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a384fbb-4972-422d-bcc9-6b7033639596-crio-socket\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.721713 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.721712    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a384fbb-4972-422d-bcc9-6b7033639596-data-volume\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.721988 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.721730    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.721988 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.721779    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.721988 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.721810    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqwl\" (UniqueName: \"kubernetes.io/projected/4a384fbb-4972-422d-bcc9-6b7033639596-kube-api-access-9xqwl\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.822873 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822774    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a384fbb-4972-422d-bcc9-6b7033639596-crio-socket\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.822873 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822818    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a384fbb-4972-422d-bcc9-6b7033639596-data-volume\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.822873 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822840    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.822873 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822871    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822896    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqwl\" (UniqueName: \"kubernetes.io/projected/4a384fbb-4972-422d-bcc9-6b7033639596-kube-api-access-9xqwl\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.822908    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a384fbb-4972-422d-bcc9-6b7033639596-crio-socket\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:38.822923    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:38.822986    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:38.823014    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:39.322996296 +0000 UTC m=+278.893929007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : configmap "kube-rbac-proxy" not found
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:38.823030    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:39.323024333 +0000 UTC m=+278.893957025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:38.823186 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.823142    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a384fbb-4972-422d-bcc9-6b7033639596-data-volume\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:38.833770 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:38.833746    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqwl\" (UniqueName: \"kubernetes.io/projected/4a384fbb-4972-422d-bcc9-6b7033639596-kube-api-access-9xqwl\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:39.326031 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:39.326000    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:39.326215 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:39.326078    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:39.326215 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:39.326138    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:59:39.326215 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:39.326161    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:39.326215 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:39.326200    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:40.326185864 +0000 UTC m=+279.897118556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : configmap "kube-rbac-proxy" not found
Sep 09 19:59:39.326407 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:39.326224    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:40.326212037 +0000 UTC m=+279.897144728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:40.333238 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:40.333196    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:40.333688 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:40.333245    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:40.333688 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:40.333339    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:59:40.333688 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:40.333381    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:40.333688 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:40.333408    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:42.333392969 +0000 UTC m=+281.904325660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : configmap "kube-rbac-proxy" not found
Sep 09 19:59:40.333688 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:40.333439    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:42.333422634 +0000 UTC m=+281.904355347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:41.102815 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:41.102788    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2297_0c874116-2f48-4afd-92dd-caf462ac9684/dns-node-resolver/0.log"
Sep 09 19:59:42.347387 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:42.347339    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:42.347387 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:42.347392    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:42.347797 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:42.347463    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:59:42.347797 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:42.347482    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:42.347797 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:42.347535    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:46.347514137 +0000 UTC m=+285.918446832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : configmap "kube-rbac-proxy" not found
Sep 09 19:59:42.347797 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:42.347554    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:46.347546435 +0000 UTC m=+285.918479127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:42.444155 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:42.444122    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xhjbb" podUID="01869d12-9e3b-4439-84bb-173a4f515694"
Sep 09 19:59:42.503716 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:42.503691    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-plptx_163aa480-c64c-49e6-aa38-b51aabaf7eb8/node-ca/0.log"
Sep 09 19:59:42.801437 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:42.801403    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:59:43.306217 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:43.306183    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwrn5_0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c/serve-healthcheck-canary/0.log"
Sep 09 19:59:46.173889 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:46.173843    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 19:59:46.174285 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.174000    2547 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:59:46.174285 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.174064    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls podName:01869d12-9e3b-4439-84bb-173a4f515694 nodeName:}" failed. No retries permitted until 2026-09-09 20:01:48.17404803 +0000 UTC m=+407.744980728 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls") pod "dns-default-xhjbb" (UID: "01869d12-9e3b-4439-84bb-173a4f515694") : secret "dns-default-metrics-tls" not found
Sep 09 19:59:46.376051 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:46.376019    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:46.376151 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:46.376060    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:46.376185 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.376149    2547 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:59:46.376224 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.376193    2547 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:46.376224 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.376220    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:54.376195424 +0000 UTC m=+293.947128133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : configmap "kube-rbac-proxy" not found
Sep 09 19:59:46.376317 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:46.376238    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls podName:4a384fbb-4972-422d-bcc9-6b7033639596 nodeName:}" failed. No retries permitted until 2026-09-09 19:59:54.376227716 +0000 UTC m=+293.947160409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bclv" (UID: "4a384fbb-4972-422d-bcc9-6b7033639596") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:59:54.436580 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.436527    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:54.436580 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.436588    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:54.437102 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.437057    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a384fbb-4972-422d-bcc9-6b7033639596-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:54.438921 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.438897    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a384fbb-4972-422d-bcc9-6b7033639596-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bclv\" (UID: \"4a384fbb-4972-422d-bcc9-6b7033639596\") " pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:54.455955 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.455935    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bclv"
Sep 09 19:59:54.723328 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.723235    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bclv"]
Sep 09 19:59:54.727595 ip-10-0-136-11 kubenswrapper[2547]: W0909 19:59:54.727568    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a384fbb_4972_422d_bcc9_6b7033639596.slice/crio-6688706e60101c6382d3c953ccecbcab585f891768233bc0abc9bec0481ad8d1 WatchSource:0}: Error finding container 6688706e60101c6382d3c953ccecbcab585f891768233bc0abc9bec0481ad8d1: Status 404 returned error can't find the container with id 6688706e60101c6382d3c953ccecbcab585f891768233bc0abc9bec0481ad8d1
Sep 09 19:59:54.834413 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.834385    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bclv" event={"ID":"4a384fbb-4972-422d-bcc9-6b7033639596","Type":"ContainerStarted","Data":"9a1b294ae478513f4b66a5c763dee024f5c9ba5e6ab949f8b362da18cdc1489c"}
Sep 09 19:59:54.834548 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:54.834443    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bclv" event={"ID":"4a384fbb-4972-422d-bcc9-6b7033639596","Type":"ContainerStarted","Data":"6688706e60101c6382d3c953ccecbcab585f891768233bc0abc9bec0481ad8d1"}
Sep 09 19:59:54.942639 ip-10-0-136-11 kubenswrapper[2547]: E0909 19:59:54.942591    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q9n5s" podUID="90d331aa-5052-4a95-91cc-b82896d0d9df"
Sep 09 19:59:56.842960 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:56.842912    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bclv" event={"ID":"4a384fbb-4972-422d-bcc9-6b7033639596","Type":"ContainerStarted","Data":"55909e2b2bf7cd98d85810861a4c5fb2edde31d21722011cf91ddff41b1f74a2"}
Sep 09 19:59:57.848028 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:57.847944    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bclv" event={"ID":"4a384fbb-4972-422d-bcc9-6b7033639596","Type":"ContainerStarted","Data":"b302e573beb9f081a807f94ebdf1d9d36dfeb368413fec6ab69b3113aa62cbe1"}
Sep 09 19:59:57.865885 ip-10-0-136-11 kubenswrapper[2547]: I0909 19:59:57.865838    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8bclv" podStartSLOduration=17.283026728 podStartE2EDuration="19.865824975s" totalImagesPullingTime="2.582798247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:59:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:59:57.864520863 +0000 UTC m=+297.435453593" watchObservedRunningTime="2026-09-09 19:59:57.865824975 +0000 UTC m=+297.436757689"
Sep 09 20:00:00.819345 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:00.819316    2547 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:00:03.209137 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.209105    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666d74f78c-8lxrt"]
Sep 09 20:00:03.212474 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.212457    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.214481 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.214458    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:00:03.214600 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.214458    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:00:03.214961 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.214947    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:00:03.215212 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.215195    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tbbl\""
Sep 09 20:00:03.221468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.221449    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:00:03.229790 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.229770    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d74f78c-8lxrt"]
Sep 09 20:00:03.296610 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296572    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-installation-pull-secrets\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296610 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296611    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-bound-sa-token\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296832 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296638    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-certificates\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296832 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296658    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-trusted-ca\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296832 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296706    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7wg\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-kube-api-access-zt7wg\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296832 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296751    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-image-registry-private-configuration\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.296832 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296778    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-tls\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.297010 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.296801    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/475a33e1-d95e-4062-8b03-08aadda4bfb0-ca-trust-extracted\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397157 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397125    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-installation-pull-secrets\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397157 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397153    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-bound-sa-token\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397172    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-certificates\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397189    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-trusted-ca\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397215    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7wg\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-kube-api-access-zt7wg\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397233    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-image-registry-private-configuration\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397364    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-tls\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397431    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/475a33e1-d95e-4062-8b03-08aadda4bfb0-ca-trust-extracted\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.397923 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.397899    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/475a33e1-d95e-4062-8b03-08aadda4bfb0-ca-trust-extracted\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.398203 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.398179    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-certificates\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.398282 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.398223    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/475a33e1-d95e-4062-8b03-08aadda4bfb0-trusted-ca\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.399859 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.399835    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-image-registry-private-configuration\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.399942 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.399855    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-registry-tls\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.399942 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.399835    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/475a33e1-d95e-4062-8b03-08aadda4bfb0-installation-pull-secrets\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.406877 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.406851    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-bound-sa-token\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.407269 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.407246    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7wg\" (UniqueName: \"kubernetes.io/projected/475a33e1-d95e-4062-8b03-08aadda4bfb0-kube-api-access-zt7wg\") pod \"image-registry-666d74f78c-8lxrt\" (UID: \"475a33e1-d95e-4062-8b03-08aadda4bfb0\") " pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.522035 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.521946    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.794258 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.794066    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666d74f78c-8lxrt"]
Sep 09 20:00:03.796695 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:00:03.796663    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475a33e1_d95e_4062_8b03_08aadda4bfb0.slice/crio-e9094b9a8fe19a59d23b75a72c3bd070bcfeca61bde885b8b794ac2d82fc5418 WatchSource:0}: Error finding container e9094b9a8fe19a59d23b75a72c3bd070bcfeca61bde885b8b794ac2d82fc5418: Status 404 returned error can't find the container with id e9094b9a8fe19a59d23b75a72c3bd070bcfeca61bde885b8b794ac2d82fc5418
Sep 09 20:00:03.866962 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.866933    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" event={"ID":"475a33e1-d95e-4062-8b03-08aadda4bfb0","Type":"ContainerStarted","Data":"4333e7299e01c2f075f91133431306e8f49b1e08900a64cf4700571f2cc5f6f9"}
Sep 09 20:00:03.867125 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.866984    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" event={"ID":"475a33e1-d95e-4062-8b03-08aadda4bfb0","Type":"ContainerStarted","Data":"e9094b9a8fe19a59d23b75a72c3bd070bcfeca61bde885b8b794ac2d82fc5418"}
Sep 09 20:00:03.867125 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.867013    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:03.885374 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:03.885325    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" podStartSLOduration=0.885306324 podStartE2EDuration="885.306324ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:00:03.884495079 +0000 UTC m=+303.455427813" watchObservedRunningTime="2026-09-09 20:00:03.885306324 +0000 UTC m=+303.456239030"
Sep 09 20:00:09.941423 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:09.941369    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 20:00:19.606787 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.606749    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lvxmg"]
Sep 09 20:00:19.612827 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.612795    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.614933 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.614894    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:00:19.615042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.614943    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:00:19.615042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.614900    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:00:19.615042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.615034    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gpvzs\""
Sep 09 20:00:19.615319 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.615301    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:00:19.615369 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.615342    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:00:19.615416 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.615342    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:00:19.713865 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.713827    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-textfile\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714077 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.713873    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714077 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.713894    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-wtmp\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714077 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.713996    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkjs\" (UniqueName: \"kubernetes.io/projected/6163133f-7fcf-4903-89b2-b0c04b415f0d-kube-api-access-2rkjs\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714077 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.714034    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.714095    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-root\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.714124    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-tls\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.714156    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-sys\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.714272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.714178    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-metrics-client-ca\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815546 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815512    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815572    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-root\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815594    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-tls\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815616    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-sys\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815632    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-metrics-client-ca\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815655    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-textfile\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815693    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.815727 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815724    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-wtmp\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815726    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-root\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815738    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-sys\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815753    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkjs\" (UniqueName: \"kubernetes.io/projected/6163133f-7fcf-4903-89b2-b0c04b415f0d-kube-api-access-2rkjs\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.815890    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-wtmp\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816229 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.816206    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-textfile\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816344 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.816324    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-metrics-client-ca\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.816389 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.816325    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-accelerators-collector-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.817946 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.817927    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.818063 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.818046    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6163133f-7fcf-4903-89b2-b0c04b415f0d-node-exporter-tls\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.826328 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.826285    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkjs\" (UniqueName: \"kubernetes.io/projected/6163133f-7fcf-4903-89b2-b0c04b415f0d-kube-api-access-2rkjs\") pod \"node-exporter-lvxmg\" (UID: \"6163133f-7fcf-4903-89b2-b0c04b415f0d\") " pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.922019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.921991    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lvxmg"
Sep 09 20:00:19.934324 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:00:19.934276    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6163133f_7fcf_4903_89b2_b0c04b415f0d.slice/crio-817bfe07f287a55c8d970496b7fffa70805b8ed53d58a90edbfc1b7d2d75fb58 WatchSource:0}: Error finding container 817bfe07f287a55c8d970496b7fffa70805b8ed53d58a90edbfc1b7d2d75fb58: Status 404 returned error can't find the container with id 817bfe07f287a55c8d970496b7fffa70805b8ed53d58a90edbfc1b7d2d75fb58
Sep 09 20:00:19.936236 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:19.936213    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:00:20.918474 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:20.918423    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvxmg" event={"ID":"6163133f-7fcf-4903-89b2-b0c04b415f0d","Type":"ContainerStarted","Data":"817bfe07f287a55c8d970496b7fffa70805b8ed53d58a90edbfc1b7d2d75fb58"}
Sep 09 20:00:22.925425 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:22.925390    2547 generic.go:356] "Generic (PLEG): container finished" podID="6163133f-7fcf-4903-89b2-b0c04b415f0d" containerID="a2a25e65b923ac07fd1d47cf3f86f32580ca39f42418a26f003f95e2cae12cd9" exitCode=0
Sep 09 20:00:22.925786 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:22.925438    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvxmg" event={"ID":"6163133f-7fcf-4903-89b2-b0c04b415f0d","Type":"ContainerDied","Data":"a2a25e65b923ac07fd1d47cf3f86f32580ca39f42418a26f003f95e2cae12cd9"}
Sep 09 20:00:23.525937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:23.525895    2547 patch_prober.go:28] interesting pod/image-registry-666d74f78c-8lxrt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:00:23.526095 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:23.525950    2547 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" podUID="475a33e1-d95e-4062-8b03-08aadda4bfb0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:00:23.930597 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:23.930563    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvxmg" event={"ID":"6163133f-7fcf-4903-89b2-b0c04b415f0d","Type":"ContainerStarted","Data":"94e631802a389c048725906ca939dcdb992880c8bba113f5a510491f57ccb029"}
Sep 09 20:00:23.930971 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:23.930617    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lvxmg" event={"ID":"6163133f-7fcf-4903-89b2-b0c04b415f0d","Type":"ContainerStarted","Data":"f6b948bc88f088d401ff04807119cff5b01cffd599540f901a2abd40a4a88e5d"}
Sep 09 20:00:23.949009 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:23.948954    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lvxmg" podStartSLOduration=2.903672087 podStartE2EDuration="4.948936607s" totalImagesPullingTime="1.984449542s" totalInitContainerRuntime="60.814978ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:00:23.947885699 +0000 UTC m=+323.518818424" watchObservedRunningTime="2026-09-09 20:00:23.948936607 +0000 UTC m=+323.519869320"
Sep 09 20:00:24.874117 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:24.874084    2547 patch_prober.go:28] interesting pod/image-registry-666d74f78c-8lxrt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:00:24.874305 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:24.874138    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" podUID="475a33e1-d95e-4062-8b03-08aadda4bfb0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:00:31.116895 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:31.116856    2547 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" podUID="ee7541f0-bee8-4b77-98d3-1c83898f2fc1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:00:33.526220 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:33.526176    2547 patch_prober.go:28] interesting pod/image-registry-666d74f78c-8lxrt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 20:00:33.526599 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:33.526241    2547 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt" podUID="475a33e1-d95e-4062-8b03-08aadda4bfb0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:00:34.873976 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:34.873946    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666d74f78c-8lxrt"
Sep 09 20:00:41.117027 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:41.116983    2547 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" podUID="ee7541f0-bee8-4b77-98d3-1c83898f2fc1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:00:51.116454 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:51.116411    2547 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" podUID="ee7541f0-bee8-4b77-98d3-1c83898f2fc1" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:00:51.116812 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:51.116479    2547 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q"
Sep 09 20:00:51.116981 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:51.116947    2547 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6c502d1a3175321d60110d840e137069aa64a765a57f72d4eecdd4ec2e9679e2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 20:00:51.117022 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:51.117002    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" podUID="ee7541f0-bee8-4b77-98d3-1c83898f2fc1" containerName="service-proxy" containerID="cri-o://6c502d1a3175321d60110d840e137069aa64a765a57f72d4eecdd4ec2e9679e2" gracePeriod=30
Sep 09 20:00:52.014781 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:52.014745    2547 generic.go:356] "Generic (PLEG): container finished" podID="ee7541f0-bee8-4b77-98d3-1c83898f2fc1" containerID="6c502d1a3175321d60110d840e137069aa64a765a57f72d4eecdd4ec2e9679e2" exitCode=2
Sep 09 20:00:52.015079 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:52.014819    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerDied","Data":"6c502d1a3175321d60110d840e137069aa64a765a57f72d4eecdd4ec2e9679e2"}
Sep 09 20:00:52.015079 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:00:52.014874    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dbdbbcf4f-rws4q" event={"ID":"ee7541f0-bee8-4b77-98d3-1c83898f2fc1","Type":"ContainerStarted","Data":"8c241a5fff4167283e0f6b885e9c6d4bbe342aa2195a53a8d001dce494e7484f"}
Sep 09 20:01:13.815592 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:13.815496    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 20:01:13.817802 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:13.817773    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d331aa-5052-4a95-91cc-b82896d0d9df-metrics-certs\") pod \"network-metrics-daemon-q9n5s\" (UID: \"90d331aa-5052-4a95-91cc-b82896d0d9df\") " pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 20:01:13.843797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:13.843769    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cpmqx\""
Sep 09 20:01:13.852179 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:13.852156    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q9n5s"
Sep 09 20:01:14.120959 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:14.120927    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q9n5s"]
Sep 09 20:01:14.123602 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:01:14.123577    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d331aa_5052_4a95_91cc_b82896d0d9df.slice/crio-c2bba1a5346804aa3ba04cfd2b65594490bbfbdc28f708d7b176dcb8cffa6318 WatchSource:0}: Error finding container c2bba1a5346804aa3ba04cfd2b65594490bbfbdc28f708d7b176dcb8cffa6318: Status 404 returned error can't find the container with id c2bba1a5346804aa3ba04cfd2b65594490bbfbdc28f708d7b176dcb8cffa6318
Sep 09 20:01:15.085629 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:15.085590    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9n5s" event={"ID":"90d331aa-5052-4a95-91cc-b82896d0d9df","Type":"ContainerStarted","Data":"c2bba1a5346804aa3ba04cfd2b65594490bbfbdc28f708d7b176dcb8cffa6318"}
Sep 09 20:01:17.094702 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:17.094662    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9n5s" event={"ID":"90d331aa-5052-4a95-91cc-b82896d0d9df","Type":"ContainerStarted","Data":"7eb0d2c9161a23e4b2329a384389cabc84b65bb3ac18f22f4881f20cf430fcf5"}
Sep 09 20:01:18.099493 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:18.099457    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q9n5s" event={"ID":"90d331aa-5052-4a95-91cc-b82896d0d9df","Type":"ContainerStarted","Data":"42060c0b72dd544995e55426650590a1ed6eecbf7b64542badba8ede4620fb3d"}
Sep 09 20:01:18.115700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:18.115643    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q9n5s" podStartSLOduration=374.25993341 podStartE2EDuration="6m17.11562784s" totalImagesPullingTime="2.85569443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:18.114972985 +0000 UTC m=+377.685905723" watchObservedRunningTime="2026-09-09 20:01:18.11562784 +0000 UTC m=+377.686560553"
Sep 09 20:01:45.802211 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:01:45.802167    2547 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xhjbb" podUID="01869d12-9e3b-4439-84bb-173a4f515694"
Sep 09 20:01:46.189550 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:46.189520    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:01:48.260445 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:48.260401    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:01:48.262669 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:48.262642    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01869d12-9e3b-4439-84bb-173a4f515694-metrics-tls\") pod \"dns-default-xhjbb\" (UID: \"01869d12-9e3b-4439-84bb-173a4f515694\") " pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:01:48.296089 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:48.296060    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4z7jm\""
Sep 09 20:01:48.301064 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:48.301045    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:01:48.570406 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:48.570322    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xhjbb"]
Sep 09 20:01:48.573608 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:01:48.573578    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01869d12_9e3b_4439_84bb_173a4f515694.slice/crio-1b041589d08b54cd44d45d3093f6d3d84f7ff784c28833f80971dd1374e87f3c WatchSource:0}: Error finding container 1b041589d08b54cd44d45d3093f6d3d84f7ff784c28833f80971dd1374e87f3c: Status 404 returned error can't find the container with id 1b041589d08b54cd44d45d3093f6d3d84f7ff784c28833f80971dd1374e87f3c
Sep 09 20:01:49.198998 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:49.198964    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xhjbb" event={"ID":"01869d12-9e3b-4439-84bb-173a4f515694","Type":"ContainerStarted","Data":"1b041589d08b54cd44d45d3093f6d3d84f7ff784c28833f80971dd1374e87f3c"}
Sep 09 20:01:52.210854 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:52.210777    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xhjbb" event={"ID":"01869d12-9e3b-4439-84bb-173a4f515694","Type":"ContainerStarted","Data":"a7fa72d56cc9f39e3fb7cd58b5753ae69e52ddb910b583b28ce8237dc3469175"}
Sep 09 20:01:52.210854 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:52.210834    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xhjbb" event={"ID":"01869d12-9e3b-4439-84bb-173a4f515694","Type":"ContainerStarted","Data":"8466cc12bd7678cacea1a6357b94eb63f3e57bbcea92eaf16a1b5d4b1f8ee9c1"}
Sep 09 20:01:52.211247 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:52.210866    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:01:52.229095 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:01:52.229051    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xhjbb" podStartSLOduration=372.935184321 podStartE2EDuration="6m16.229037367s" totalImagesPullingTime="3.293853046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:52.228410941 +0000 UTC m=+411.799343675" watchObservedRunningTime="2026-09-09 20:01:52.229037367 +0000 UTC m=+411.799970080"
Sep 09 20:02:02.216912 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:02:02.216870    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xhjbb"
Sep 09 20:07:16.711663 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.711582    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"]
Sep 09 20:07:16.714591 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.714575    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.716796 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.716769    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:07:16.716796 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.716787    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:07:16.716925 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.716787    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-z4d2j\""
Sep 09 20:07:16.717074 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.717057    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:07:16.727204 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.727184    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"]
Sep 09 20:07:16.788042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.788010    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.788188 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.788057    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchkj\" (UniqueName: \"kubernetes.io/projected/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-kube-api-access-lchkj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.889345 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.889317    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchkj\" (UniqueName: \"kubernetes.io/projected/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-kube-api-access-lchkj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.889446 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.889375    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.891620 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.891594    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:16.903879 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:16.903858    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchkj\" (UniqueName: \"kubernetes.io/projected/5ea3f567-24f8-46b1-9e27-8d36f7a105ce-kube-api-access-lchkj\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-mvmgv\" (UID: \"5ea3f567-24f8-46b1-9e27-8d36f7a105ce\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:17.024152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:17.024068    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:17.298422 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:17.298277    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"]
Sep 09 20:07:17.303249 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:07:17.303214    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea3f567_24f8_46b1_9e27_8d36f7a105ce.slice/crio-2ebe950f84e190de21cdd75a5d215a4fef789939363c24d757a6259c9565eec0 WatchSource:0}: Error finding container 2ebe950f84e190de21cdd75a5d215a4fef789939363c24d757a6259c9565eec0: Status 404 returned error can't find the container with id 2ebe950f84e190de21cdd75a5d215a4fef789939363c24d757a6259c9565eec0
Sep 09 20:07:17.305692 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:17.305673    2547 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:07:18.252886 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:18.252844    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv" event={"ID":"5ea3f567-24f8-46b1-9e27-8d36f7a105ce","Type":"ContainerStarted","Data":"2ebe950f84e190de21cdd75a5d215a4fef789939363c24d757a6259c9565eec0"}
Sep 09 20:07:21.270894 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:21.270858    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv" event={"ID":"5ea3f567-24f8-46b1-9e27-8d36f7a105ce","Type":"ContainerStarted","Data":"3f835f44cd3e7814b7b0492ba92f3b7cef1f965c59fca66932f6f0c1e70a23a0"}
Sep 09 20:07:21.271247 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:21.270978    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:21.292352 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:21.292281    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv" podStartSLOduration=1.832204345 podStartE2EDuration="5.292267334s" totalImagesPullingTime="3.460062989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:21.291364574 +0000 UTC m=+740.862297309" watchObservedRunningTime="2026-09-09 20:07:21.292267334 +0000 UTC m=+740.863200047"
Sep 09 20:07:22.596767 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.596729    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-b8pb2"]
Sep 09 20:07:22.599830 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.599813    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.602194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.602172    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:07:22.602313 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.602224    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-65zq4\""
Sep 09 20:07:22.602313 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.602257    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 09 20:07:22.608225 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.608197    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-b8pb2"]
Sep 09 20:07:22.634532 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.634502    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67b2\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-kube-api-access-r67b2\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.634681 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.634563    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-certificates\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.735876 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.735840    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-certificates\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.736049 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.735892    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67b2\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-kube-api-access-r67b2\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.738330 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.738307    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-certificates\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.745280 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.745254    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67b2\" (UniqueName: \"kubernetes.io/projected/e59b4206-8fe1-4537-a665-5e2bfaf7482a-kube-api-access-r67b2\") pod \"keda-admission-6c4d864b86-b8pb2\" (UID: \"e59b4206-8fe1-4537-a665-5e2bfaf7482a\") " pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:22.910908 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:22.910870    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:23.183336 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:23.183227    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-b8pb2"]
Sep 09 20:07:23.186581 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:07:23.186549    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59b4206_8fe1_4537_a665_5e2bfaf7482a.slice/crio-f39be1ef9c63ee6e7709a5f828fd6f22ad2ed1cbf8e29d7c5415679b0eefaca0 WatchSource:0}: Error finding container f39be1ef9c63ee6e7709a5f828fd6f22ad2ed1cbf8e29d7c5415679b0eefaca0: Status 404 returned error can't find the container with id f39be1ef9c63ee6e7709a5f828fd6f22ad2ed1cbf8e29d7c5415679b0eefaca0
Sep 09 20:07:23.278558 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:23.278511    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2" event={"ID":"e59b4206-8fe1-4537-a665-5e2bfaf7482a","Type":"ContainerStarted","Data":"f39be1ef9c63ee6e7709a5f828fd6f22ad2ed1cbf8e29d7c5415679b0eefaca0"}
Sep 09 20:07:25.288418 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:25.288378    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2" event={"ID":"e59b4206-8fe1-4537-a665-5e2bfaf7482a","Type":"ContainerStarted","Data":"96add9d635bcd67dea43b1f1c1669a846aa5b075569c12fa7ce88db0e14a5b0c"}
Sep 09 20:07:25.288809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:25.288521    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:07:25.305543 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:25.305354    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2" podStartSLOduration=1.649853319 podStartE2EDuration="3.30533313s" totalImagesPullingTime="1.655479811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:25.304662168 +0000 UTC m=+744.875594882" watchObservedRunningTime="2026-09-09 20:07:25.30533313 +0000 UTC m=+744.876265876"
Sep 09 20:07:42.276609 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:42.276581    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-mvmgv"
Sep 09 20:07:46.295107 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:07:46.295077    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-b8pb2"
Sep 09 20:08:43.254151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.254116    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-7xgws"]
Sep 09 20:08:43.257159 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.257143    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.260104 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.260083    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:08:43.260454 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.260427    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:08:43.260454 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.260441    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:08:43.260587 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.260441    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v5gh2\""
Sep 09 20:08:43.267795 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.267775    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7xgws"]
Sep 09 20:08:43.288443 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.288414    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5pfj\" (UniqueName: \"kubernetes.io/projected/c180b994-b76b-4c36-b001-8208ee3e999a-kube-api-access-k5pfj\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.288557 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.288472    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c180b994-b76b-4c36-b001-8208ee3e999a-data\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.388879 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.388845    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c180b994-b76b-4c36-b001-8208ee3e999a-data\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.389014 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.388912    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5pfj\" (UniqueName: \"kubernetes.io/projected/c180b994-b76b-4c36-b001-8208ee3e999a-kube-api-access-k5pfj\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.389192 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.389175    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c180b994-b76b-4c36-b001-8208ee3e999a-data\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.396449 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.396423    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5pfj\" (UniqueName: \"kubernetes.io/projected/c180b994-b76b-4c36-b001-8208ee3e999a-kube-api-access-k5pfj\") pod \"seaweedfs-64568bcd49-7xgws\" (UID: \"c180b994-b76b-4c36-b001-8208ee3e999a\") " pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.565773 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.565705    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:43.834353 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:43.834321    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7xgws"]
Sep 09 20:08:43.836927 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:08:43.836891    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc180b994_b76b_4c36_b001_8208ee3e999a.slice/crio-d021b475bb2d4c03224e14aa59ac00cf27100503e1097ae21cffba95b8fd544e WatchSource:0}: Error finding container d021b475bb2d4c03224e14aa59ac00cf27100503e1097ae21cffba95b8fd544e: Status 404 returned error can't find the container with id d021b475bb2d4c03224e14aa59ac00cf27100503e1097ae21cffba95b8fd544e
Sep 09 20:08:44.563785 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:44.563741    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7xgws" event={"ID":"c180b994-b76b-4c36-b001-8208ee3e999a","Type":"ContainerStarted","Data":"d021b475bb2d4c03224e14aa59ac00cf27100503e1097ae21cffba95b8fd544e"}
Sep 09 20:08:47.575242 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:47.575205    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7xgws" event={"ID":"c180b994-b76b-4c36-b001-8208ee3e999a","Type":"ContainerStarted","Data":"67bd65a9b760815667463478e58be3d6b46a61ee5d3dc3bc0f1338fccff56fa2"}
Sep 09 20:08:47.575657 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:47.575270    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:08:47.592621 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:47.592569    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-7xgws" podStartSLOduration=1.7246537910000002 podStartE2EDuration="4.592553984s" totalImagesPullingTime="2.867900193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:47.59131026 +0000 UTC m=+827.162242975" watchObservedRunningTime="2026-09-09 20:08:47.592553984 +0000 UTC m=+827.163486698"
Sep 09 20:08:53.581455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:08:53.581425    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-7xgws"
Sep 09 20:10:02.501097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.501061    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jpkkk"]
Sep 09 20:10:02.503436 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.503420    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.506328 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.506309    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tczb6\""
Sep 09 20:10:02.506475 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.506458    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:10:02.516394 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.516358    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jpkkk"]
Sep 09 20:10:02.524697 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.524672    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-z894z"]
Sep 09 20:10:02.526805 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.526789    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.529007 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.528982    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:10:02.529298 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.529254    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:10:02.530797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.530751    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-72lck\""
Sep 09 20:10:02.541701 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.541678    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-z894z"]
Sep 09 20:10:02.641087 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.641045    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5011f8-6539-4059-ae7b-e49ee3bafaea-tls-certs\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.641087 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.641087    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntrh\" (UniqueName: \"kubernetes.io/projected/f6688563-e9ec-4edb-99c2-8d115a3e67d2-kube-api-access-sntrh\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.641367 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.641125    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.641367 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.641149    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-cert\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.641367 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.641224    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4g4g\" (UniqueName: \"kubernetes.io/projected/3d5011f8-6539-4059-ae7b-e49ee3bafaea-kube-api-access-p4g4g\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.742626 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.742576    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5011f8-6539-4059-ae7b-e49ee3bafaea-tls-certs\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.742626 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.742637    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntrh\" (UniqueName: \"kubernetes.io/projected/f6688563-e9ec-4edb-99c2-8d115a3e67d2-kube-api-access-sntrh\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.742900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.742680    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.742900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.742706    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-cert\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.742900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.742773    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4g4g\" (UniqueName: \"kubernetes.io/projected/3d5011f8-6539-4059-ae7b-e49ee3bafaea-kube-api-access-p4g4g\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.742900 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:10:02.742814    2547 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 20:10:02.742900 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:10:02.742898    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs podName:f6688563-e9ec-4edb-99c2-8d115a3e67d2 nodeName:}" failed. No retries permitted until 2026-09-09 20:10:03.242879571 +0000 UTC m=+902.813812271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs") pod "odh-model-controller-7b966d7984-z894z" (UID: "f6688563-e9ec-4edb-99c2-8d115a3e67d2") : secret "odh-model-controller-metrics-tls" not found
Sep 09 20:10:02.745140 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.745116    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d5011f8-6539-4059-ae7b-e49ee3bafaea-tls-certs\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.745247 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.745204    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-cert\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.751141 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.751078    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4g4g\" (UniqueName: \"kubernetes.io/projected/3d5011f8-6539-4059-ae7b-e49ee3bafaea-kube-api-access-p4g4g\") pod \"model-serving-api-7b954ffc95-jpkkk\" (UID: \"3d5011f8-6539-4059-ae7b-e49ee3bafaea\") " pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:02.751272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.751254    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntrh\" (UniqueName: \"kubernetes.io/projected/f6688563-e9ec-4edb-99c2-8d115a3e67d2-kube-api-access-sntrh\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:02.812354 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:02.812308    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:03.094797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.094737    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jpkkk"]
Sep 09 20:10:03.097336 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:10:03.097309    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5011f8_6539_4059_ae7b_e49ee3bafaea.slice/crio-dd233f7c4f6c15d24a353e07694d3e4289bef93792a337fd490baa480d6952f8 WatchSource:0}: Error finding container dd233f7c4f6c15d24a353e07694d3e4289bef93792a337fd490baa480d6952f8: Status 404 returned error can't find the container with id dd233f7c4f6c15d24a353e07694d3e4289bef93792a337fd490baa480d6952f8
Sep 09 20:10:03.246039 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.246007    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:03.248304 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.248273    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6688563-e9ec-4edb-99c2-8d115a3e67d2-metrics-certs\") pod \"odh-model-controller-7b966d7984-z894z\" (UID: \"f6688563-e9ec-4edb-99c2-8d115a3e67d2\") " pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:03.437091 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.437058    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:03.708282 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.704827    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-z894z"]
Sep 09 20:10:03.711718 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:10:03.711689    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6688563_e9ec_4edb_99c2_8d115a3e67d2.slice/crio-63fa0b2eb74e826f17d02f5dc4f446a11b896c989539eb91aa3b0530994d68bf WatchSource:0}: Error finding container 63fa0b2eb74e826f17d02f5dc4f446a11b896c989539eb91aa3b0530994d68bf: Status 404 returned error can't find the container with id 63fa0b2eb74e826f17d02f5dc4f446a11b896c989539eb91aa3b0530994d68bf
Sep 09 20:10:03.831538 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.831499    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jpkkk" event={"ID":"3d5011f8-6539-4059-ae7b-e49ee3bafaea","Type":"ContainerStarted","Data":"dd233f7c4f6c15d24a353e07694d3e4289bef93792a337fd490baa480d6952f8"}
Sep 09 20:10:03.832626 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:03.832600    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-z894z" event={"ID":"f6688563-e9ec-4edb-99c2-8d115a3e67d2","Type":"ContainerStarted","Data":"63fa0b2eb74e826f17d02f5dc4f446a11b896c989539eb91aa3b0530994d68bf"}
Sep 09 20:10:05.842894 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:05.842808    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jpkkk" event={"ID":"3d5011f8-6539-4059-ae7b-e49ee3bafaea","Type":"ContainerStarted","Data":"17bd6a1830163e2fa55423e3dc16f258a5a4e53eb0e628d62a7b41b554233fcc"}
Sep 09 20:10:05.843329 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:05.842968    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:05.871575 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:05.871519    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jpkkk" podStartSLOduration=1.5578087059999999 podStartE2EDuration="3.871503856s" totalImagesPullingTime="2.31369515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:05.870555193 +0000 UTC m=+905.441487934" watchObservedRunningTime="2026-09-09 20:10:05.871503856 +0000 UTC m=+905.442436570"
Sep 09 20:10:06.848506 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:06.848468    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-z894z" event={"ID":"f6688563-e9ec-4edb-99c2-8d115a3e67d2","Type":"ContainerStarted","Data":"b9bfac54f37e08ad2373dcb8f7c4862be41f59e0ae2657eaec0ad226a0f699c4"}
Sep 09 20:10:06.848980 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:06.848567    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:10:06.865793 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:06.865745    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-z894z" podStartSLOduration=1.915589764 podStartE2EDuration="4.865727911s" totalImagesPullingTime="2.950138147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:06.864350211 +0000 UTC m=+906.435282938" watchObservedRunningTime="2026-09-09 20:10:06.865727911 +0000 UTC m=+906.436660659"
Sep 09 20:10:16.853039 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:16.853006    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jpkkk"
Sep 09 20:10:17.854964 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:10:17.854928    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-z894z"
Sep 09 20:11:50.565054 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.565022    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:11:50.568265 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.568250    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.570228 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.570204    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-w6n8h\""
Sep 09 20:11:50.570423 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.570409    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:11:50.570635 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.570613    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3b763-predictor-serving-cert\""
Sep 09 20:11:50.570771 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.570756    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3b763-kube-rbac-proxy-sar-config\""
Sep 09 20:11:50.570985 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.570967    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:11:50.581032 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.581007    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:11:50.730863 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.730826    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvp2\" (UniqueName: \"kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.730863 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.730864    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.731088 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.730959    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.775697 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.775661    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:11:50.778794 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.778776    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:50.780645 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.780622    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3b763-kube-rbac-proxy-sar-config\""
Sep 09 20:11:50.780757 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.780735    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3b763-predictor-serving-cert\""
Sep 09 20:11:50.788893 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.788871    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:11:50.832218 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.832145    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvp2\" (UniqueName: \"kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.832218 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.832175    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.832218 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.832215    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.832514 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:11:50.832329    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3b763-predictor-serving-cert: secret "success-200-isvc-3b763-predictor-serving-cert" not found
Sep 09 20:11:50.832514 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:11:50.832411    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls podName:bf756f90-33aa-462b-93c7-852055fbb991 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:51.332388889 +0000 UTC m=+1010.903321582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls") pod "success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" (UID: "bf756f90-33aa-462b-93c7-852055fbb991") : secret "success-200-isvc-3b763-predictor-serving-cert" not found
Sep 09 20:11:50.832826 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.832808    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.845910 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.845877    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvp2\" (UniqueName: \"kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:50.933243 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.933206    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:50.933420 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.933267    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:50.933420 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:50.933315    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdkj\" (UniqueName: \"kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.034685 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.034651    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.034859 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.034697    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdkj\" (UniqueName: \"kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.034859 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.034755    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.035466 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.035442    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.037103 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.037082    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.045505 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.045479    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdkj\" (UniqueName: \"kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj\") pod \"error-404-isvc-3b763-predictor-786696fb7c-k98n4\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") " pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.089126 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.088898    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:11:51.337931 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.337895    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:51.340448 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.340385    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") pod \"success-200-isvc-3b763-predictor-84bbbdf8d-mj65k\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") " pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:51.365064 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.365040    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:11:51.367634 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:11:51.367603    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a1cff0_78d5_4f5f_ace4_9fd633444128.slice/crio-0671580391afa64cd219fcf82063ea21e32024e8070faf1083c6ba68b861a0f8 WatchSource:0}: Error finding container 0671580391afa64cd219fcf82063ea21e32024e8070faf1083c6ba68b861a0f8: Status 404 returned error can't find the container with id 0671580391afa64cd219fcf82063ea21e32024e8070faf1083c6ba68b861a0f8
Sep 09 20:11:51.478136 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.478101    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:11:51.747857 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:51.747828    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:11:51.751253 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:11:51.751229    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf756f90_33aa_462b_93c7_852055fbb991.slice/crio-da1c89d04223c49c366a8304ca142ce789776950976d463909355a6eb976b7ca WatchSource:0}: Error finding container da1c89d04223c49c366a8304ca142ce789776950976d463909355a6eb976b7ca: Status 404 returned error can't find the container with id da1c89d04223c49c366a8304ca142ce789776950976d463909355a6eb976b7ca
Sep 09 20:11:52.239758 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:52.239610    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerStarted","Data":"0671580391afa64cd219fcf82063ea21e32024e8070faf1083c6ba68b861a0f8"}
Sep 09 20:11:52.243894 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:11:52.243803    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerStarted","Data":"da1c89d04223c49c366a8304ca142ce789776950976d463909355a6eb976b7ca"}
Sep 09 20:12:08.334651 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:08.334611    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerStarted","Data":"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"}
Sep 09 20:12:08.337073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:08.337035    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerStarted","Data":"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"}
Sep 09 20:12:11.351612 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.351571    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerStarted","Data":"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"}
Sep 09 20:12:11.352076 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.351735    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:12:11.353405 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.353375    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerStarted","Data":"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"}
Sep 09 20:12:11.353540 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.353502    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:12:11.353586 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.353549    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:12:11.354934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.354909    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:11.369972 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.369921    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podStartSLOduration=2.4741396030000002 podStartE2EDuration="21.369906562s" totalImagesPullingTime="18.895766959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:11.369278126 +0000 UTC m=+1030.940210840" watchObservedRunningTime="2026-09-09 20:12:11.369906562 +0000 UTC m=+1030.940839274"
Sep 09 20:12:11.394942 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:11.394896    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podStartSLOduration=2.883380404 podStartE2EDuration="21.394877325s" totalImagesPullingTime="18.511496921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:11.393599588 +0000 UTC m=+1030.964532301" watchObservedRunningTime="2026-09-09 20:12:11.394877325 +0000 UTC m=+1030.965810040"
Sep 09 20:12:12.357675 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:12.357630    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:12.358126 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:12.357742    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:12:12.359015 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:12.358985    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:13.360805 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:13.360766    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:17.362575 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:17.362547    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:12:17.363090 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:17.363064    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:18.365716 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:18.365688    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:12:18.366238 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:18.366213    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:27.363189 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:27.363141    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:28.366766 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:28.366720    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:37.363106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:37.363069    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:38.366695 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:38.366650    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:47.363083 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:47.363045    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:12:48.366756 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:48.366718    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 09 20:12:57.364450 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:57.364372    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:12:58.367002 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:12:58.366975    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:13:24.638965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.638935    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:13:24.642122 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.642101    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.644524 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.644503    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2727-predictor-serving-cert\""
Sep 09 20:13:24.644968 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.644948    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d2727-kube-rbac-proxy-sar-config\""
Sep 09 20:13:24.656750 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.656724    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:13:24.667591 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.667569    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:13:24.667815 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.667776    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" containerID="cri-o://80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e" gracePeriod=30
Sep 09 20:13:24.667878 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.667821    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kube-rbac-proxy" containerID="cri-o://7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3" gracePeriod=30
Sep 09 20:13:24.682844 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.682818    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxpp\" (UniqueName: \"kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.682949 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.682857    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.683013 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.682985    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.767338 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.767311    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:13:24.770340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.770322    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.772218 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.772199    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d2727-predictor-serving-cert\""
Sep 09 20:13:24.772503 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.772488    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d2727-kube-rbac-proxy-sar-config\""
Sep 09 20:13:24.782265 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.782245    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:13:24.783951 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.783933    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.784022 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.783970    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxpp\" (UniqueName: \"kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.784022 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.783988    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.784660 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.784640    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.786569 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.786549    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.797232 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.797204    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxpp\" (UniqueName: \"kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp\") pod \"success-200-isvc-d2727-predictor-775569db54-5zrm6\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") " pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.820635 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.820604    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:13:24.820938 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.820891    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kube-rbac-proxy" containerID="cri-o://464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331" gracePeriod=30
Sep 09 20:13:24.821029 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.820883    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container" containerID="cri-o://8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5" gracePeriod=30
Sep 09 20:13:24.885110 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.885084    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.885221 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.885133    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pk7\" (UniqueName: \"kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.885221 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.885152    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.953104 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.953073    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:24.986519 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.986491    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.986665 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.986565    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pk7\" (UniqueName: \"kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.986665 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.986585    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.986786 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:24.986664    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d2727-predictor-serving-cert: secret "error-404-isvc-d2727-predictor-serving-cert" not found
Sep 09 20:13:24.986786 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:24.986734    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls podName:71cbc44b-aea9-4c53-85c7-7d03e61d0e42 nodeName:}" failed. No retries permitted until 2026-09-09 20:13:25.486713508 +0000 UTC m=+1105.057646217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls") pod "error-404-isvc-d2727-predictor-6487c46c67-cphk6" (UID: "71cbc44b-aea9-4c53-85c7-7d03e61d0e42") : secret "error-404-isvc-d2727-predictor-serving-cert" not found
Sep 09 20:13:24.987144 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.987126    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:24.998254 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:24.998039    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pk7\" (UniqueName: \"kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:25.219668 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.219476    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:13:25.222134 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:13:25.222108    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b3ca7a_0aeb_4dd4_bc64_f66f7c8abfe9.slice/crio-1a0a9afd450b42eb26037a3af002b27da40d93a52b472d7600183a07554b92d5 WatchSource:0}: Error finding container 1a0a9afd450b42eb26037a3af002b27da40d93a52b472d7600183a07554b92d5: Status 404 returned error can't find the container with id 1a0a9afd450b42eb26037a3af002b27da40d93a52b472d7600183a07554b92d5
Sep 09 20:13:25.488641 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.488554    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:25.490853 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.490834    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") pod \"error-404-isvc-d2727-predictor-6487c46c67-cphk6\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") " pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:25.622990 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.622956    2547 generic.go:356] "Generic (PLEG): container finished" podID="bf756f90-33aa-462b-93c7-852055fbb991" containerID="7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3" exitCode=2
Sep 09 20:13:25.623156 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.623026    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerDied","Data":"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"}
Sep 09 20:13:25.624967 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.624941    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerStarted","Data":"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"}
Sep 09 20:13:25.625112 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.625096    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:25.625196 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.625146    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:25.625196 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.625175    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerStarted","Data":"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"}
Sep 09 20:13:25.625332 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.625206    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerStarted","Data":"1a0a9afd450b42eb26037a3af002b27da40d93a52b472d7600183a07554b92d5"}
Sep 09 20:13:25.626533 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.626506    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:13:25.626870 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.626854    2547 generic.go:356] "Generic (PLEG): container finished" podID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerID="464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331" exitCode=2
Sep 09 20:13:25.626944 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.626899    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerDied","Data":"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"}
Sep 09 20:13:25.642337 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.642253    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podStartSLOduration=1.642237696 podStartE2EDuration="1.642237696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:25.641082887 +0000 UTC m=+1105.212015614" watchObservedRunningTime="2026-09-09 20:13:25.642237696 +0000 UTC m=+1105.213170410"
Sep 09 20:13:25.680714 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.680683    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:25.950937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:25.950906    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:13:25.953852 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:13:25.953829    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cbc44b_aea9_4c53_85c7_7d03e61d0e42.slice/crio-db51209f048338afbb93b17f2abfa2cc97e7255e6eeb8dd1a0aa992c63712c00 WatchSource:0}: Error finding container db51209f048338afbb93b17f2abfa2cc97e7255e6eeb8dd1a0aa992c63712c00: Status 404 returned error can't find the container with id db51209f048338afbb93b17f2abfa2cc97e7255e6eeb8dd1a0aa992c63712c00
Sep 09 20:13:26.632470 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632428    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerStarted","Data":"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"}
Sep 09 20:13:26.632645 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632496    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerStarted","Data":"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"}
Sep 09 20:13:26.632645 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632625    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:26.632775 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632658    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:26.632775 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632675    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerStarted","Data":"db51209f048338afbb93b17f2abfa2cc97e7255e6eeb8dd1a0aa992c63712c00"}
Sep 09 20:13:26.632950 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.632924    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:13:26.634076 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.634055    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:13:26.650687 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:26.650626    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podStartSLOduration=2.6506089790000003 podStartE2EDuration="2.650608979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:26.64818355 +0000 UTC m=+1106.219116290" watchObservedRunningTime="2026-09-09 20:13:26.650608979 +0000 UTC m=+1106.221541706"
Sep 09 20:13:27.358718 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:27.358677    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 09 20:13:27.363007 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:27.362985    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 09 20:13:27.636608 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:27.636565    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:13:27.906454 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:27.906432    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:13:28.009586 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.009552    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") pod \"bf756f90-33aa-462b-93c7-852055fbb991\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") "
Sep 09 20:13:28.009586 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.009593    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqvp2\" (UniqueName: \"kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2\") pod \"bf756f90-33aa-462b-93c7-852055fbb991\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") "
Sep 09 20:13:28.009844 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.009633    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config\") pod \"bf756f90-33aa-462b-93c7-852055fbb991\" (UID: \"bf756f90-33aa-462b-93c7-852055fbb991\") "
Sep 09 20:13:28.010026 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.010003    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf756f90-33aa-462b-93c7-852055fbb991" (UID: "bf756f90-33aa-462b-93c7-852055fbb991"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:28.011823 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.011798    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf756f90-33aa-462b-93c7-852055fbb991" (UID: "bf756f90-33aa-462b-93c7-852055fbb991"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:28.011918 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.011816    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2" (OuterVolumeSpecName: "kube-api-access-sqvp2") pod "bf756f90-33aa-462b-93c7-852055fbb991" (UID: "bf756f90-33aa-462b-93c7-852055fbb991"). InnerVolumeSpecName "kube-api-access-sqvp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:28.110799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.110771    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf756f90-33aa-462b-93c7-852055fbb991-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.110799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.110794    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqvp2\" (UniqueName: \"kubernetes.io/projected/bf756f90-33aa-462b-93c7-852055fbb991-kube-api-access-sqvp2\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.110951 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.110804    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf756f90-33aa-462b-93c7-852055fbb991-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.248420 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.248392    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:13:28.312371 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.312338    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config\") pod \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") "
Sep 09 20:13:28.312560 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.312409    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfdkj\" (UniqueName: \"kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj\") pod \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") "
Sep 09 20:13:28.312560 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.312450    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls\") pod \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\" (UID: \"a3a1cff0-78d5-4f5f-ace4-9fd633444128\") "
Sep 09 20:13:28.312755 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.312732    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3a1cff0-78d5-4f5f-ace4-9fd633444128" (UID: "a3a1cff0-78d5-4f5f-ace4-9fd633444128"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:13:28.314548 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.314515    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj" (OuterVolumeSpecName: "kube-api-access-bfdkj") pod "a3a1cff0-78d5-4f5f-ace4-9fd633444128" (UID: "a3a1cff0-78d5-4f5f-ace4-9fd633444128"). InnerVolumeSpecName "kube-api-access-bfdkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:13:28.314548 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.314524    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3a1cff0-78d5-4f5f-ace4-9fd633444128" (UID: "a3a1cff0-78d5-4f5f-ace4-9fd633444128"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:13:28.417786 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.413531    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3a1cff0-78d5-4f5f-ace4-9fd633444128-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.417968 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.417803    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.417968 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.417821    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdkj\" (UniqueName: \"kubernetes.io/projected/a3a1cff0-78d5-4f5f-ace4-9fd633444128-kube-api-access-bfdkj\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:13:28.641862 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.641826    2547 generic.go:356] "Generic (PLEG): container finished" podID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerID="8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5" exitCode=0
Sep 09 20:13:28.642033 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.641891    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"
Sep 09 20:13:28.642033 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.641920    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerDied","Data":"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"}
Sep 09 20:13:28.642144 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.642037    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4" event={"ID":"a3a1cff0-78d5-4f5f-ace4-9fd633444128","Type":"ContainerDied","Data":"0671580391afa64cd219fcf82063ea21e32024e8070faf1083c6ba68b861a0f8"}
Sep 09 20:13:28.642144 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.642079    2547 scope.go:117] "RemoveContainer" containerID="464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"
Sep 09 20:13:28.644042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.644015    2547 generic.go:356] "Generic (PLEG): container finished" podID="bf756f90-33aa-462b-93c7-852055fbb991" containerID="80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e" exitCode=0
Sep 09 20:13:28.644151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.644061    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerDied","Data":"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"}
Sep 09 20:13:28.644151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.644085    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"
Sep 09 20:13:28.644151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.644112    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k" event={"ID":"bf756f90-33aa-462b-93c7-852055fbb991","Type":"ContainerDied","Data":"da1c89d04223c49c366a8304ca142ce789776950976d463909355a6eb976b7ca"}
Sep 09 20:13:28.653956 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.653858    2547 scope.go:117] "RemoveContainer" containerID="8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"
Sep 09 20:13:28.662098 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.662082    2547 scope.go:117] "RemoveContainer" containerID="464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"
Sep 09 20:13:28.662363 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:28.662344    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331\": container with ID starting with 464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331 not found: ID does not exist" containerID="464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"
Sep 09 20:13:28.662413 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.662372    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331"} err="failed to get container status \"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331\": rpc error: code = NotFound desc = could not find container \"464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331\": container with ID starting with 464a0ed463158072ba87a0550a03b066044578cc17a17ad3bc76b7e7d96ff331 not found: ID does not exist"
Sep 09 20:13:28.662413 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.662389    2547 scope.go:117] "RemoveContainer" containerID="8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"
Sep 09 20:13:28.662644 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:28.662629    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5\": container with ID starting with 8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5 not found: ID does not exist" containerID="8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"
Sep 09 20:13:28.662684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.662647    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5"} err="failed to get container status \"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5\": rpc error: code = NotFound desc = could not find container \"8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5\": container with ID starting with 8bf09cc60414ef5691354b09a6fcc33f41a1f61f88d240c34ac4d440b2735ff5 not found: ID does not exist"
Sep 09 20:13:28.662684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.662659    2547 scope.go:117] "RemoveContainer" containerID="7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"
Sep 09 20:13:28.666923 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.666900    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:13:28.669708 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.669688    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3b763-predictor-786696fb7c-k98n4"]
Sep 09 20:13:28.670263 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.670250    2547 scope.go:117] "RemoveContainer" containerID="80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"
Sep 09 20:13:28.677214 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.677196    2547 scope.go:117] "RemoveContainer" containerID="7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"
Sep 09 20:13:28.677521 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:28.677501    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3\": container with ID starting with 7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3 not found: ID does not exist" containerID="7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"
Sep 09 20:13:28.677567 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.677528    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3"} err="failed to get container status \"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3\": rpc error: code = NotFound desc = could not find container \"7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3\": container with ID starting with 7244b1858c210e580a2c400827ee882c06b8a99b9073160cd907d1f45fc3d7d3 not found: ID does not exist"
Sep 09 20:13:28.677567 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.677546    2547 scope.go:117] "RemoveContainer" containerID="80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"
Sep 09 20:13:28.677781 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:13:28.677767    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e\": container with ID starting with 80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e not found: ID does not exist" containerID="80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"
Sep 09 20:13:28.677826 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.677784    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e"} err="failed to get container status \"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e\": rpc error: code = NotFound desc = could not find container \"80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e\": container with ID starting with 80e1efa66d3a340ee699572398933eb9131c523a39dd4162a3bd93c13153f16e not found: ID does not exist"
Sep 09 20:13:28.677954 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.677937    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:13:28.681934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.681915    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3b763-predictor-84bbbdf8d-mj65k"]
Sep 09 20:13:28.947706 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.947622    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" path="/var/lib/kubelet/pods/a3a1cff0-78d5-4f5f-ace4-9fd633444128/volumes"
Sep 09 20:13:28.948153 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:28.948008    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf756f90-33aa-462b-93c7-852055fbb991" path="/var/lib/kubelet/pods/bf756f90-33aa-462b-93c7-852055fbb991/volumes"
Sep 09 20:13:31.638091 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:31.638058    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:13:31.638570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:31.638480    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:13:32.640657 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:32.640630    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:13:32.641119 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:32.641077    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:13:41.639506 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:41.639469    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:13:42.641263 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:42.641226    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:13:51.638974 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:51.638930    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:13:52.641655 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:13:52.641609    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:14:01.638958 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:01.638920    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:14:02.641655 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:02.641615    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 09 20:14:10.694455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694414    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:14:10.694900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694850    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kube-rbac-proxy"
Sep 09 20:14:10.694900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694868    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kube-rbac-proxy"
Sep 09 20:14:10.694900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694883    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container"
Sep 09 20:14:10.694900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694890    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container"
Sep 09 20:14:10.694900 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694901    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694910    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694931    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kube-rbac-proxy"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.694941    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kube-rbac-proxy"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.695010    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kserve-container"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.695024    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf756f90-33aa-462b-93c7-852055fbb991" containerName="kube-rbac-proxy"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.695032    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kube-rbac-proxy"
Sep 09 20:14:10.695152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.695041    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a1cff0-78d5-4f5f-ace4-9fd633444128" containerName="kserve-container"
Sep 09 20:14:10.699273 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.699251    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.701185 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.701167    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e942-predictor-serving-cert\""
Sep 09 20:14:10.701300 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.701168    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2e942-kube-rbac-proxy-sar-config\""
Sep 09 20:14:10.706785 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.706766    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:14:10.741096 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.741062    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6v6n\" (UniqueName: \"kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.741238 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.741113    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.741238 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.741145    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.783952 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.783925    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:14:10.787131 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.787116    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.788884 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.788861    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e942-kube-rbac-proxy-sar-config\""
Sep 09 20:14:10.789135 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.789120    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2e942-predictor-serving-cert\""
Sep 09 20:14:10.795397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.795371    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:14:10.842357 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842323    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvrv\" (UniqueName: \"kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.842537 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842371    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6v6n\" (UniqueName: \"kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.842537 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842424    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.842537 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842455    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.842537 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842499    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.842740 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.842567    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.842740 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:14:10.842576    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-2e942-predictor-serving-cert: secret "success-200-isvc-2e942-predictor-serving-cert" not found
Sep 09 20:14:10.842740 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:14:10.842666    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls podName:4be82fbf-6585-47da-abbb-c8ef329043ec nodeName:}" failed. No retries permitted until 2026-09-09 20:14:11.342649007 +0000 UTC m=+1150.913581699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls") pod "success-200-isvc-2e942-predictor-6569cb6498-x7nfr" (UID: "4be82fbf-6585-47da-abbb-c8ef329043ec") : secret "success-200-isvc-2e942-predictor-serving-cert" not found
Sep 09 20:14:10.843164 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.843138    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.850320 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.850268    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6v6n\" (UniqueName: \"kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:10.942931 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.942888    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.943131 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.942952    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.943131 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.942977    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvrv\" (UniqueName: \"kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.943131 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:14:10.943101    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2e942-predictor-serving-cert: secret "error-404-isvc-2e942-predictor-serving-cert" not found
Sep 09 20:14:10.943313 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:14:10.943178    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls podName:c407662e-cdc1-446c-9904-02069a00ab7a nodeName:}" failed. No retries permitted until 2026-09-09 20:14:11.443157686 +0000 UTC m=+1151.014090393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls") pod "error-404-isvc-2e942-predictor-7cb698b899-cbq9c" (UID: "c407662e-cdc1-446c-9904-02069a00ab7a") : secret "error-404-isvc-2e942-predictor-serving-cert" not found
Sep 09 20:14:10.943569 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.943548    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:10.951806 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:10.951749    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvrv\" (UniqueName: \"kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:11.345701 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.345605    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:11.348003 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.347973    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") pod \"success-200-isvc-2e942-predictor-6569cb6498-x7nfr\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") " pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:11.446816 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.446779    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:11.449177 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.449149    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") pod \"error-404-isvc-2e942-predictor-7cb698b899-cbq9c\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") " pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:11.609215 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.609111    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:11.639457 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.639424    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:14:11.696622 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.696592    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:11.886938 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.886906    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:14:11.889160 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:14:11.889132    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be82fbf_6585_47da_abbb_c8ef329043ec.slice/crio-4aae8d465081d9819c64bada5a43694edaa35661df0ba3e8bbc05d88e363161e WatchSource:0}: Error finding container 4aae8d465081d9819c64bada5a43694edaa35661df0ba3e8bbc05d88e363161e: Status 404 returned error can't find the container with id 4aae8d465081d9819c64bada5a43694edaa35661df0ba3e8bbc05d88e363161e
Sep 09 20:14:11.983143 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:11.983109    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:14:11.986324 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:14:11.986266    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc407662e_cdc1_446c_9904_02069a00ab7a.slice/crio-1e372d3694d156daf75d3df4886a2c6e34b5601dc5af96b1abae7f228a656cb2 WatchSource:0}: Error finding container 1e372d3694d156daf75d3df4886a2c6e34b5601dc5af96b1abae7f228a656cb2: Status 404 returned error can't find the container with id 1e372d3694d156daf75d3df4886a2c6e34b5601dc5af96b1abae7f228a656cb2
Sep 09 20:14:12.642021 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.641976    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:14:12.819355 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.819322    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerStarted","Data":"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"}
Sep 09 20:14:12.819809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.819395    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:12.819809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.819430    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerStarted","Data":"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"}
Sep 09 20:14:12.819809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.819579    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:12.819809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.819644    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerStarted","Data":"4aae8d465081d9819c64bada5a43694edaa35661df0ba3e8bbc05d88e363161e"}
Sep 09 20:14:12.821180 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821147    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:12.821391 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821370    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerStarted","Data":"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"}
Sep 09 20:14:12.821447 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821422    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerStarted","Data":"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"}
Sep 09 20:14:12.821447 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821446    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:12.821524 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821464    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerStarted","Data":"1e372d3694d156daf75d3df4886a2c6e34b5601dc5af96b1abae7f228a656cb2"}
Sep 09 20:14:12.821524 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.821483    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:12.822532 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.822507    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:12.842936 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.842877    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podStartSLOduration=2.842858172 podStartE2EDuration="2.842858172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:12.841474058 +0000 UTC m=+1152.412406787" watchObservedRunningTime="2026-09-09 20:14:12.842858172 +0000 UTC m=+1152.413790901"
Sep 09 20:14:12.866525 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:12.866475    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podStartSLOduration=2.866460653 podStartE2EDuration="2.866460653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:12.865364448 +0000 UTC m=+1152.436297181" watchObservedRunningTime="2026-09-09 20:14:12.866460653 +0000 UTC m=+1152.437393367"
Sep 09 20:14:13.825513 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:13.825479    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:13.825937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:13.825482    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:18.829516 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:18.829480    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:18.830438 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:18.830401    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:18.835445 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:18.835426    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:14:18.835922 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:18.835891    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:28.830661 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:28.830582    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:28.835797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:28.835771    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:38.831053 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:38.831014    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:38.836335 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:38.836285    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:48.831243 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:48.831205    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 09 20:14:48.836492 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:48.836457    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:14:58.831001 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:58.830970    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:14:58.836441 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:14:58.836412    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:22:39.376081 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.376044    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:22:39.378468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.376352    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" containerID="cri-o://a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6" gracePeriod=30
Sep 09 20:22:39.378468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.376429    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kube-rbac-proxy" containerID="cri-o://e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582" gracePeriod=30
Sep 09 20:22:39.431437 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.431401    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:22:39.434655 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.434627    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.436935 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.436912    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e029b-predictor-serving-cert\""
Sep 09 20:22:39.437084 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.437032    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e029b-kube-rbac-proxy-sar-config\""
Sep 09 20:22:39.447371 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.447347    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:22:39.471969 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.471934    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:22:39.472236 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.472203    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container" containerID="cri-o://436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e" gracePeriod=30
Sep 09 20:22:39.472395 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.472271    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kube-rbac-proxy" containerID="cri-o://8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba" gracePeriod=30
Sep 09 20:22:39.476074 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.476048    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.476184 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.476101    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.476184 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.476161    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zp7x\" (UniqueName: \"kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.516930 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.516902    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:22:39.519839 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.519824    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.521551 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.521532    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e029b-kube-rbac-proxy-sar-config\""
Sep 09 20:22:39.521747 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.521732    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e029b-predictor-serving-cert\""
Sep 09 20:22:39.537024 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.537005    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:22:39.577253 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577231    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zp7x\" (UniqueName: \"kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.577395 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577271    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.577446 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577393    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.577446 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577442    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.577516 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577467    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvss\" (UniqueName: \"kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.577516 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.577494    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.577615 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:39.577597    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e029b-predictor-serving-cert: secret "success-200-isvc-e029b-predictor-serving-cert" not found
Sep 09 20:22:39.577675 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:39.577665    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls podName:a934850a-3086-4391-b93c-95b96ea4bdb9 nodeName:}" failed. No retries permitted until 2026-09-09 20:22:40.077644717 +0000 UTC m=+1659.648577423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls") pod "success-200-isvc-e029b-predictor-6b5b586b98-847mn" (UID: "a934850a-3086-4391-b93c-95b96ea4bdb9") : secret "success-200-isvc-e029b-predictor-serving-cert" not found
Sep 09 20:22:39.578026 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.578009    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.587863 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.587840    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zp7x\" (UniqueName: \"kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:39.678378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.678284    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.678500 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.678379    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvss\" (UniqueName: \"kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.678500 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.678443    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.678903 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.678882    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.680703 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.680676    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.686566 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.686548    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvss\" (UniqueName: \"kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss\") pod \"error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") " pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.829409 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.829372    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:39.878933 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.878790    2547 generic.go:356] "Generic (PLEG): container finished" podID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerID="e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582" exitCode=2
Sep 09 20:22:39.878933 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.878858    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerDied","Data":"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"}
Sep 09 20:22:39.881721 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.881569    2547 generic.go:356] "Generic (PLEG): container finished" podID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerID="8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba" exitCode=2
Sep 09 20:22:39.881721 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:39.881652    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerDied","Data":"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"}
Sep 09 20:22:40.081831 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.081743    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:40.084149 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.084125    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") pod \"success-200-isvc-e029b-predictor-6b5b586b98-847mn\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") " pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:40.100239 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.100219    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:22:40.102595 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:22:40.102571    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce218e6_084f_4bb2_adca_e1bd24b173c8.slice/crio-f23782f7992fa7905172b2b1d689f4df6a544b3fdf8326c1b93555296a119521 WatchSource:0}: Error finding container f23782f7992fa7905172b2b1d689f4df6a544b3fdf8326c1b93555296a119521: Status 404 returned error can't find the container with id f23782f7992fa7905172b2b1d689f4df6a544b3fdf8326c1b93555296a119521
Sep 09 20:22:40.345948 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.345849    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:40.620271 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.620249    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:22:40.621993 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:22:40.621967    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda934850a_3086_4391_b93c_95b96ea4bdb9.slice/crio-5c171269d9332f8ce4716c3dce8ffbe3ceade93fb8cdcf5568bd6b51dacaa1d1 WatchSource:0}: Error finding container 5c171269d9332f8ce4716c3dce8ffbe3ceade93fb8cdcf5568bd6b51dacaa1d1: Status 404 returned error can't find the container with id 5c171269d9332f8ce4716c3dce8ffbe3ceade93fb8cdcf5568bd6b51dacaa1d1
Sep 09 20:22:40.887738 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.887647    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerStarted","Data":"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"}
Sep 09 20:22:40.887738 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.887721    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerStarted","Data":"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"}
Sep 09 20:22:40.887981 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.887814    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:40.887981 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.887878    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:40.887981 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.887906    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerStarted","Data":"f23782f7992fa7905172b2b1d689f4df6a544b3fdf8326c1b93555296a119521"}
Sep 09 20:22:40.888954 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.888926    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:22:40.889663 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.889640    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerStarted","Data":"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"}
Sep 09 20:22:40.894312 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.890046    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerStarted","Data":"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"}
Sep 09 20:22:40.894312 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.890195    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerStarted","Data":"5c171269d9332f8ce4716c3dce8ffbe3ceade93fb8cdcf5568bd6b51dacaa1d1"}
Sep 09 20:22:40.894312 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.890248    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:40.894312 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.890306    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:40.894999 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.894875    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:22:40.910192 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.910150    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podStartSLOduration=1.910135621 podStartE2EDuration="1.910135621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:40.908761652 +0000 UTC m=+1660.479694373" watchObservedRunningTime="2026-09-09 20:22:40.910135621 +0000 UTC m=+1660.481068332"
Sep 09 20:22:40.926912 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:40.926873    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podStartSLOduration=1.926860106 podStartE2EDuration="1.926860106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:40.926160177 +0000 UTC m=+1660.497092910" watchObservedRunningTime="2026-09-09 20:22:40.926860106 +0000 UTC m=+1660.497792820"
Sep 09 20:22:41.633923 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:41.633884    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 09 20:22:41.639056 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:41.639016    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 09 20:22:41.894693 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:41.894607    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:22:41.894841 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:41.894608    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:22:42.514734 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.514704    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:22:42.600444 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.600373    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls\") pod \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") "
Sep 09 20:22:42.600444 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.600441    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config\") pod \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") "
Sep 09 20:22:42.600649 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.600485    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldxpp\" (UniqueName: \"kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp\") pod \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\" (UID: \"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9\") "
Sep 09 20:22:42.600887 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.600855    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" (UID: "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:42.602578 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.602549    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" (UID: "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:42.602696 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.602644    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp" (OuterVolumeSpecName: "kube-api-access-ldxpp") pod "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" (UID: "73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9"). InnerVolumeSpecName "kube-api-access-ldxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:42.616024 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.616005    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:22:42.701981 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.701948    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4pk7\" (UniqueName: \"kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7\") pod \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") "
Sep 09 20:22:42.702397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702002    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") pod \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") "
Sep 09 20:22:42.702397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702048    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config\") pod \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\" (UID: \"71cbc44b-aea9-4c53-85c7-7d03e61d0e42\") "
Sep 09 20:22:42.702397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702259    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.702397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702278    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.702397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702318    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxpp\" (UniqueName: \"kubernetes.io/projected/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9-kube-api-access-ldxpp\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.702567 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.702439    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71cbc44b-aea9-4c53-85c7-7d03e61d0e42" (UID: "71cbc44b-aea9-4c53-85c7-7d03e61d0e42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:42.703937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.703917    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71cbc44b-aea9-4c53-85c7-7d03e61d0e42" (UID: "71cbc44b-aea9-4c53-85c7-7d03e61d0e42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:42.703982 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.703949    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7" (OuterVolumeSpecName: "kube-api-access-h4pk7") pod "71cbc44b-aea9-4c53-85c7-7d03e61d0e42" (UID: "71cbc44b-aea9-4c53-85c7-7d03e61d0e42"). InnerVolumeSpecName "kube-api-access-h4pk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:42.802684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.802651    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pk7\" (UniqueName: \"kubernetes.io/projected/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-api-access-h4pk7\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.802684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.802678    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.802684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.802688    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71cbc44b-aea9-4c53-85c7-7d03e61d0e42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:22:42.899737 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.899707    2547 generic.go:356] "Generic (PLEG): container finished" podID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerID="a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6" exitCode=0
Sep 09 20:22:42.899893 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.899761    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"
Sep 09 20:22:42.899893 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.899792    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerDied","Data":"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"}
Sep 09 20:22:42.899893 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.899860    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6" event={"ID":"73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9","Type":"ContainerDied","Data":"1a0a9afd450b42eb26037a3af002b27da40d93a52b472d7600183a07554b92d5"}
Sep 09 20:22:42.900006 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.899905    2547 scope.go:117] "RemoveContainer" containerID="e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"
Sep 09 20:22:42.901538 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.901515    2547 generic.go:356] "Generic (PLEG): container finished" podID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerID="436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e" exitCode=0
Sep 09 20:22:42.901657 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.901556    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"
Sep 09 20:22:42.901657 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.901567    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerDied","Data":"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"}
Sep 09 20:22:42.901657 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.901605    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6" event={"ID":"71cbc44b-aea9-4c53-85c7-7d03e61d0e42","Type":"ContainerDied","Data":"db51209f048338afbb93b17f2abfa2cc97e7255e6eeb8dd1a0aa992c63712c00"}
Sep 09 20:22:42.908475 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.908275    2547 scope.go:117] "RemoveContainer" containerID="a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"
Sep 09 20:22:42.916074 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.916055    2547 scope.go:117] "RemoveContainer" containerID="e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"
Sep 09 20:22:42.916365 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:42.916346    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582\": container with ID starting with e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582 not found: ID does not exist" containerID="e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"
Sep 09 20:22:42.916414 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.916374    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582"} err="failed to get container status \"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582\": rpc error: code = NotFound desc = could not find container \"e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582\": container with ID starting with e852b62c3be16511443cc4d819686ea4c2d78e196a1dbbc5b02f86c7937a4582 not found: ID does not exist"
Sep 09 20:22:42.916414 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.916389    2547 scope.go:117] "RemoveContainer" containerID="a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"
Sep 09 20:22:42.916610 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:42.916595    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6\": container with ID starting with a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6 not found: ID does not exist" containerID="a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"
Sep 09 20:22:42.916649 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.916615    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6"} err="failed to get container status \"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6\": rpc error: code = NotFound desc = could not find container \"a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6\": container with ID starting with a75c28779826f7cff6f0da0f8dcb20d6df055bb36823ddba2313e0a03b94d9f6 not found: ID does not exist"
Sep 09 20:22:42.916649 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.916627    2547 scope.go:117] "RemoveContainer" containerID="8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"
Sep 09 20:22:42.920588 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.920568    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:22:42.923618 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.923602    2547 scope.go:117] "RemoveContainer" containerID="436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"
Sep 09 20:22:42.926465 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.926446    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d2727-predictor-775569db54-5zrm6"]
Sep 09 20:22:42.930581 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.930567    2547 scope.go:117] "RemoveContainer" containerID="8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"
Sep 09 20:22:42.930829 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:42.930809    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba\": container with ID starting with 8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba not found: ID does not exist" containerID="8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"
Sep 09 20:22:42.930880 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.930845    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba"} err="failed to get container status \"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba\": rpc error: code = NotFound desc = could not find container \"8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba\": container with ID starting with 8e3c1e728701bb7819e82139be85faac49d9a475bb560db5e3cdec07d90b75ba not found: ID does not exist"
Sep 09 20:22:42.930880 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.930861    2547 scope.go:117] "RemoveContainer" containerID="436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"
Sep 09 20:22:42.931093 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:22:42.931077    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e\": container with ID starting with 436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e not found: ID does not exist" containerID="436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"
Sep 09 20:22:42.931147 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.931099    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e"} err="failed to get container status \"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e\": rpc error: code = NotFound desc = could not find container \"436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e\": container with ID starting with 436e92f0a8bdbba33ec115e3e91f8d03a44dae3a4bd2e189a60a9b10bac1996e not found: ID does not exist"
Sep 09 20:22:42.936613 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.936592    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:22:42.940604 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.940584    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d2727-predictor-6487c46c67-cphk6"]
Sep 09 20:22:42.948816 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.948788    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" path="/var/lib/kubelet/pods/71cbc44b-aea9-4c53-85c7-7d03e61d0e42/volumes"
Sep 09 20:22:42.949387 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:42.949368    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" path="/var/lib/kubelet/pods/73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9/volumes"
Sep 09 20:22:46.899519 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:46.899491    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:22:46.899918 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:46.899855    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:22:46.899918 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:46.899891    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:22:46.900240 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:46.900216    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:22:56.900238 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:56.900203    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:22:56.900721 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:22:56.900372    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:23:06.900936 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:06.900899    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:23:06.901357 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:06.900900    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:23:16.900217 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:16.900175    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:23:16.900736 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:16.900272    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:23:25.237985 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.237905    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:23:25.238449 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.238117    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container" containerID="cri-o://9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd" gracePeriod=30
Sep 09 20:23:25.238449 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.238155    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kube-rbac-proxy" containerID="cri-o://9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9" gracePeriod=30
Sep 09 20:23:25.272620 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.272591    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:23:25.272849 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.272806    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container" containerID="cri-o://6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5" gracePeriod=30
Sep 09 20:23:25.272932 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.272824    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kube-rbac-proxy" containerID="cri-o://e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235" gracePeriod=30
Sep 09 20:23:25.363983 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.363954    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:23:25.364277 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364264    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kube-rbac-proxy"
Sep 09 20:23:25.364348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364278    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kube-rbac-proxy"
Sep 09 20:23:25.364348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364318    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container"
Sep 09 20:23:25.364348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364325    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container"
Sep 09 20:23:25.364348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364332    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container"
Sep 09 20:23:25.364348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364338    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364357    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kube-rbac-proxy"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364362    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kube-rbac-proxy"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364425    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kube-rbac-proxy"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364433    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kserve-container"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364439    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="71cbc44b-aea9-4c53-85c7-7d03e61d0e42" containerName="kserve-container"
Sep 09 20:23:25.364535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.364445    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b3ca7a-0aeb-4dd4-bc64-f66f7c8abfe9" containerName="kube-rbac-proxy"
Sep 09 20:23:25.367215 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.367201    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.369009 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.368985    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bf2e8-kube-rbac-proxy-sar-config\""
Sep 09 20:23:25.369212 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.369198    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bf2e8-predictor-serving-cert\""
Sep 09 20:23:25.376100 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.376079    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:23:25.463333 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.463300    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:23:25.466564 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.466549    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.468329 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.468283    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bf2e8-kube-rbac-proxy-sar-config\""
Sep 09 20:23:25.468521 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.468508    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bf2e8-predictor-serving-cert\""
Sep 09 20:23:25.476029 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.476009    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:23:25.523745 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.523654    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.523745 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.523694    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7mkl\" (UniqueName: \"kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.523934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.523787    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.624655 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624624    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65gvm\" (UniqueName: \"kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.624811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624663    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.624811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624689    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7mkl\" (UniqueName: \"kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.624811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624742    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.624811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624801    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.625000 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.624861    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.625483 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.625464    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.627079 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.627054    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.633004 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.632982    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7mkl\" (UniqueName: \"kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl\") pod \"success-200-isvc-bf2e8-predictor-c577cf68-h9ggr\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") " pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.676883 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.676852    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:25.726548 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.726154    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.726548 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.726248    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65gvm\" (UniqueName: \"kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.726692 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.726550    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.727345 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.727058    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.729577 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.729532    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.734868 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.734842    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65gvm\" (UniqueName: \"kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm\") pod \"error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") " pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.776812 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.776748    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:25.946267 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:25.946236    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:23:25.949460 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:23:25.949425    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58895f95_0127_4cde_bf11_7e427cbcd906.slice/crio-8ea99d9f8929bc85ff70d79a57303550453dd47553df57a6612b09750cd0bc96 WatchSource:0}: Error finding container 8ea99d9f8929bc85ff70d79a57303550453dd47553df57a6612b09750cd0bc96: Status 404 returned error can't find the container with id 8ea99d9f8929bc85ff70d79a57303550453dd47553df57a6612b09750cd0bc96
Sep 09 20:23:26.048881 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.048847    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:23:26.052226 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:23:26.052201    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ceac87_95c5_40e6_a9b1_d13275442279.slice/crio-cbd812866d07640a05c71ea4a2efafc9b3f81b3e7026c05c6fbfc434a204f951 WatchSource:0}: Error finding container cbd812866d07640a05c71ea4a2efafc9b3f81b3e7026c05c6fbfc434a204f951: Status 404 returned error can't find the container with id cbd812866d07640a05c71ea4a2efafc9b3f81b3e7026c05c6fbfc434a204f951
Sep 09 20:23:26.084180 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.084153    2547 generic.go:356] "Generic (PLEG): container finished" podID="c407662e-cdc1-446c-9904-02069a00ab7a" containerID="e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235" exitCode=2
Sep 09 20:23:26.084332 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.084218    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerDied","Data":"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"}
Sep 09 20:23:26.086091 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.086068    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerStarted","Data":"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"}
Sep 09 20:23:26.086201 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.086118    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerStarted","Data":"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"}
Sep 09 20:23:26.086201 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.086149    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerStarted","Data":"8ea99d9f8929bc85ff70d79a57303550453dd47553df57a6612b09750cd0bc96"}
Sep 09 20:23:26.086347 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.086319    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:26.086474 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.086461    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:26.087998 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.087968    2547 generic.go:356] "Generic (PLEG): container finished" podID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerID="9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9" exitCode=2
Sep 09 20:23:26.088086 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.088011    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerDied","Data":"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"}
Sep 09 20:23:26.088086 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.088071    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:23:26.089271 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.089250    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerStarted","Data":"cbd812866d07640a05c71ea4a2efafc9b3f81b3e7026c05c6fbfc434a204f951"}
Sep 09 20:23:26.106269 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.106224    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podStartSLOduration=1.10620556 podStartE2EDuration="1.10620556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:26.104546774 +0000 UTC m=+1705.675479514" watchObservedRunningTime="2026-09-09 20:23:26.10620556 +0000 UTC m=+1705.677138275"
Sep 09 20:23:26.900499 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.900464    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:23:26.901303 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:26.901264    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:23:27.095713 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:27.095669    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerStarted","Data":"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"}
Sep 09 20:23:27.095918 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:27.095739    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerStarted","Data":"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"}
Sep 09 20:23:27.096085 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:27.096055    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:23:28.099524 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.099498    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:28.100107 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.099572    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:28.100779 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.100757    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:23:28.706319 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.706284    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:23:28.708820 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.708801    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:23:28.724938 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.724890    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podStartSLOduration=3.724875576 podStartE2EDuration="3.724875576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:27.122371005 +0000 UTC m=+1706.693303738" watchObservedRunningTime="2026-09-09 20:23:28.724875576 +0000 UTC m=+1708.295808289"
Sep 09 20:23:28.854551 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854517    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") pod \"4be82fbf-6585-47da-abbb-c8ef329043ec\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") "
Sep 09 20:23:28.854742 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854591    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztvrv\" (UniqueName: \"kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv\") pod \"c407662e-cdc1-446c-9904-02069a00ab7a\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") "
Sep 09 20:23:28.854742 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854611    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6v6n\" (UniqueName: \"kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n\") pod \"4be82fbf-6585-47da-abbb-c8ef329043ec\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") "
Sep 09 20:23:28.854742 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854633    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config\") pod \"c407662e-cdc1-446c-9904-02069a00ab7a\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") "
Sep 09 20:23:28.854742 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854651    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") pod \"c407662e-cdc1-446c-9904-02069a00ab7a\" (UID: \"c407662e-cdc1-446c-9904-02069a00ab7a\") "
Sep 09 20:23:28.854742 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.854669    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config\") pod \"4be82fbf-6585-47da-abbb-c8ef329043ec\" (UID: \"4be82fbf-6585-47da-abbb-c8ef329043ec\") "
Sep 09 20:23:28.855125 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.855079    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c407662e-cdc1-446c-9904-02069a00ab7a" (UID: "c407662e-cdc1-446c-9904-02069a00ab7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:28.855125 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.855102    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4be82fbf-6585-47da-abbb-c8ef329043ec" (UID: "4be82fbf-6585-47da-abbb-c8ef329043ec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:28.856791 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.856760    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n" (OuterVolumeSpecName: "kube-api-access-z6v6n") pod "4be82fbf-6585-47da-abbb-c8ef329043ec" (UID: "4be82fbf-6585-47da-abbb-c8ef329043ec"). InnerVolumeSpecName "kube-api-access-z6v6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:28.856791 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.856778    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c407662e-cdc1-446c-9904-02069a00ab7a" (UID: "c407662e-cdc1-446c-9904-02069a00ab7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:28.857000 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.856984    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4be82fbf-6585-47da-abbb-c8ef329043ec" (UID: "4be82fbf-6585-47da-abbb-c8ef329043ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:28.857042 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.857018    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv" (OuterVolumeSpecName: "kube-api-access-ztvrv") pod "c407662e-cdc1-446c-9904-02069a00ab7a" (UID: "c407662e-cdc1-446c-9904-02069a00ab7a"). InnerVolumeSpecName "kube-api-access-ztvrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:28.955114 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955097    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6v6n\" (UniqueName: \"kubernetes.io/projected/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-api-access-z6v6n\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:28.955194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955116    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c407662e-cdc1-446c-9904-02069a00ab7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:28.955194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955125    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c407662e-cdc1-446c-9904-02069a00ab7a-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:28.955194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955135    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4be82fbf-6585-47da-abbb-c8ef329043ec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:28.955194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955143    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4be82fbf-6585-47da-abbb-c8ef329043ec-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:28.955194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:28.955152    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztvrv\" (UniqueName: \"kubernetes.io/projected/c407662e-cdc1-446c-9904-02069a00ab7a-kube-api-access-ztvrv\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:29.105394 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.105360    2547 generic.go:356] "Generic (PLEG): container finished" podID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerID="9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd" exitCode=0
Sep 09 20:23:29.105763 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.105409    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"
Sep 09 20:23:29.105763 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.105449    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerDied","Data":"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"}
Sep 09 20:23:29.105763 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.105511    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr" event={"ID":"4be82fbf-6585-47da-abbb-c8ef329043ec","Type":"ContainerDied","Data":"4aae8d465081d9819c64bada5a43694edaa35661df0ba3e8bbc05d88e363161e"}
Sep 09 20:23:29.106374 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.105542    2547 scope.go:117] "RemoveContainer" containerID="9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"
Sep 09 20:23:29.107103 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.107070    2547 generic.go:356] "Generic (PLEG): container finished" podID="c407662e-cdc1-446c-9904-02069a00ab7a" containerID="6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5" exitCode=0
Sep 09 20:23:29.107228 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.107120    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"
Sep 09 20:23:29.107228 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.107158    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerDied","Data":"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"}
Sep 09 20:23:29.107228 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.107219    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c" event={"ID":"c407662e-cdc1-446c-9904-02069a00ab7a","Type":"ContainerDied","Data":"1e372d3694d156daf75d3df4886a2c6e34b5601dc5af96b1abae7f228a656cb2"}
Sep 09 20:23:29.107809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.107659    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:23:29.114328 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.114310    2547 scope.go:117] "RemoveContainer" containerID="9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"
Sep 09 20:23:29.121174 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.121154    2547 scope.go:117] "RemoveContainer" containerID="9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"
Sep 09 20:23:29.121538 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:29.121516    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9\": container with ID starting with 9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9 not found: ID does not exist" containerID="9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"
Sep 09 20:23:29.121602 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.121549    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9"} err="failed to get container status \"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9\": rpc error: code = NotFound desc = could not find container \"9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9\": container with ID starting with 9b2acd8da38f69a6b825a77d380df824cb89ce7814118d9bad0aedff4b5090d9 not found: ID does not exist"
Sep 09 20:23:29.121602 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.121566    2547 scope.go:117] "RemoveContainer" containerID="9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"
Sep 09 20:23:29.121863 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:29.121844    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd\": container with ID starting with 9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd not found: ID does not exist" containerID="9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"
Sep 09 20:23:29.121941 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.121866    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd"} err="failed to get container status \"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd\": rpc error: code = NotFound desc = could not find container \"9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd\": container with ID starting with 9fad3dadccfc2a1fa21b31b03b5c6da6e953de25558171fd9c0912d30a6700fd not found: ID does not exist"
Sep 09 20:23:29.121941 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.121882    2547 scope.go:117] "RemoveContainer" containerID="e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"
Sep 09 20:23:29.123591 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.123568    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:23:29.127574 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.127550    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2e942-predictor-7cb698b899-cbq9c"]
Sep 09 20:23:29.129600 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.129575    2547 scope.go:117] "RemoveContainer" containerID="6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"
Sep 09 20:23:29.136509 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.136491    2547 scope.go:117] "RemoveContainer" containerID="e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"
Sep 09 20:23:29.136723 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.136704    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:23:29.136779 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:29.136750    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235\": container with ID starting with e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235 not found: ID does not exist" containerID="e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"
Sep 09 20:23:29.136779 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.136769    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235"} err="failed to get container status \"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235\": rpc error: code = NotFound desc = could not find container \"e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235\": container with ID starting with e2f5608b80362ddcd6e74762f6de92b406a7516032a1700496d21e3033ad6235 not found: ID does not exist"
Sep 09 20:23:29.136878 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.136783    2547 scope.go:117] "RemoveContainer" containerID="6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"
Sep 09 20:23:29.137035 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:29.137021    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5\": container with ID starting with 6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5 not found: ID does not exist" containerID="6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"
Sep 09 20:23:29.137070 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.137040    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5"} err="failed to get container status \"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5\": rpc error: code = NotFound desc = could not find container \"6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5\": container with ID starting with 6bf6926f68593a8c47b7aa1d49ef1f9ea1bae102eba1ee89ea29196aabe949c5 not found: ID does not exist"
Sep 09 20:23:29.139872 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:29.139852    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2e942-predictor-6569cb6498-x7nfr"]
Sep 09 20:23:30.947857 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:30.947808    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" path="/var/lib/kubelet/pods/4be82fbf-6585-47da-abbb-c8ef329043ec/volumes"
Sep 09 20:23:30.948460 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:30.948436    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" path="/var/lib/kubelet/pods/c407662e-cdc1-446c-9904-02069a00ab7a/volumes"
Sep 09 20:23:32.101847 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:32.101821    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:23:32.102317 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:32.102275    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:23:34.111802 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:34.111772    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:23:34.112200 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:34.112174    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:23:42.102633 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:42.102594    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:23:44.112755 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:44.112713    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:23:49.613013 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.612980    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:23:49.613421 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.613250    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container" containerID="cri-o://f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4" gracePeriod=30
Sep 09 20:23:49.613421 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.613307    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kube-rbac-proxy" containerID="cri-o://a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661" gracePeriod=30
Sep 09 20:23:49.649492 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.649457    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:23:49.649755 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.649727    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container" containerID="cri-o://ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c" gracePeriod=30
Sep 09 20:23:49.649862 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.649833    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kube-rbac-proxy" containerID="cri-o://b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70" gracePeriod=30
Sep 09 20:23:49.736971 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.736942    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:23:49.737265 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737252    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737267    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737275    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737281    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737316    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kube-rbac-proxy"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737323    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kube-rbac-proxy"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737329    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kube-rbac-proxy"
Sep 09 20:23:49.737340 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737335    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kube-rbac-proxy"
Sep 09 20:23:49.737615 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737401    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kube-rbac-proxy"
Sep 09 20:23:49.737615 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737409    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kserve-container"
Sep 09 20:23:49.737615 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737418    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="c407662e-cdc1-446c-9904-02069a00ab7a" containerName="kserve-container"
Sep 09 20:23:49.737615 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.737425    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be82fbf-6585-47da-abbb-c8ef329043ec" containerName="kube-rbac-proxy"
Sep 09 20:23:49.740148 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.740133    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.742838 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.742813    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c295-predictor-serving-cert\""
Sep 09 20:23:49.742949 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.742866    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3c295-kube-rbac-proxy-sar-config\""
Sep 09 20:23:49.751786 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.751756    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:23:49.812256 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.812219    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.812432 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.812268    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dzn\" (UniqueName: \"kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.812432 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.812401    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.839572 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.839542    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:23:49.842717 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.842694    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:49.844822 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.844799    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c295-predictor-serving-cert\""
Sep 09 20:23:49.844954 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.844873    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3c295-kube-rbac-proxy-sar-config\""
Sep 09 20:23:49.853714 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.853678    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:23:49.912940 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.912877    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.912940 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.912916    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:49.913106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.912949    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:49.913106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.912983    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.913106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.912999    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dzn\" (UniqueName: \"kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.913106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.913022    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxhq\" (UniqueName: \"kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:49.913565 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.913542    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.915491 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.915474    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:49.920108 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:49.920085    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dzn\" (UniqueName: \"kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn\") pod \"success-200-isvc-3c295-predictor-c97b4b754-44xk4\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") " pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:50.013477 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.013444    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.013622 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.013486    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.013622 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.013520    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxhq\" (UniqueName: \"kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.013622 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:50.013604    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3c295-predictor-serving-cert: secret "error-404-isvc-3c295-predictor-serving-cert" not found
Sep 09 20:23:50.013783 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:50.013673    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls podName:2f8c5e33-99fa-4e19-9286-eb3f5ec93e20 nodeName:}" failed. No retries permitted until 2026-09-09 20:23:50.513651498 +0000 UTC m=+1730.084584192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls") pod "error-404-isvc-3c295-predictor-6f998d6587-44pj7" (UID: "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20") : secret "error-404-isvc-3c295-predictor-serving-cert" not found
Sep 09 20:23:50.014105 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.014084    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.024569 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.024549    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxhq\" (UniqueName: \"kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.050030 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.049983    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:50.198990 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.198911    2547 generic.go:356] "Generic (PLEG): container finished" podID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerID="b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70" exitCode=2
Sep 09 20:23:50.199145 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.198989    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerDied","Data":"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"}
Sep 09 20:23:50.200808 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.200786    2547 generic.go:356] "Generic (PLEG): container finished" podID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerID="a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661" exitCode=2
Sep 09 20:23:50.200898 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.200829    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerDied","Data":"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"}
Sep 09 20:23:50.319342 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.319319    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:23:50.321674 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:23:50.321648    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989b24c7_2c45_46e5_82aa_556c7ebcdb0e.slice/crio-c9a594c0af3b3fa10b6d555c049f5564144b3d9039d0aeb3475bb5da45a1b01a WatchSource:0}: Error finding container c9a594c0af3b3fa10b6d555c049f5564144b3d9039d0aeb3475bb5da45a1b01a: Status 404 returned error can't find the container with id c9a594c0af3b3fa10b6d555c049f5564144b3d9039d0aeb3475bb5da45a1b01a
Sep 09 20:23:50.516901 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.516820    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.519116 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.519096    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") pod \"error-404-isvc-3c295-predictor-6f998d6587-44pj7\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") " pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:50.756004 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:50.755961    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:51.026266 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.026191    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:23:51.029055 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:23:51.029029    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8c5e33_99fa_4e19_9286_eb3f5ec93e20.slice/crio-b61c9f6e51d7a871e6268542ac4ebb46d188429b215f9a7a36b00ca719f91f3e WatchSource:0}: Error finding container b61c9f6e51d7a871e6268542ac4ebb46d188429b215f9a7a36b00ca719f91f3e: Status 404 returned error can't find the container with id b61c9f6e51d7a871e6268542ac4ebb46d188429b215f9a7a36b00ca719f91f3e
Sep 09 20:23:51.207379 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.207330    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerStarted","Data":"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"}
Sep 09 20:23:51.207538 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.207419    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerStarted","Data":"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"}
Sep 09 20:23:51.207538 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.207462    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerStarted","Data":"c9a594c0af3b3fa10b6d555c049f5564144b3d9039d0aeb3475bb5da45a1b01a"}
Sep 09 20:23:51.207538 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.207495    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:51.209120 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.209096    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerStarted","Data":"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"}
Sep 09 20:23:51.209229 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.209139    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerStarted","Data":"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"}
Sep 09 20:23:51.209229 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.209158    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerStarted","Data":"b61c9f6e51d7a871e6268542ac4ebb46d188429b215f9a7a36b00ca719f91f3e"}
Sep 09 20:23:51.209229 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.209179    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:51.224231 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.224185    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podStartSLOduration=2.224167078 podStartE2EDuration="2.224167078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:51.22333106 +0000 UTC m=+1730.794263792" watchObservedRunningTime="2026-09-09 20:23:51.224167078 +0000 UTC m=+1730.795099793"
Sep 09 20:23:51.238298 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.238253    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podStartSLOduration=2.238239788 podStartE2EDuration="2.238239788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:51.238131234 +0000 UTC m=+1730.809063984" watchObservedRunningTime="2026-09-09 20:23:51.238239788 +0000 UTC m=+1730.809172484"
Sep 09 20:23:51.895030 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.894991    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:23:51.895403 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:51.894991    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 09 20:23:52.103124 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:52.103085    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:23:52.213410 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:52.213314    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:52.213410 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:52.213386    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:52.214840 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:52.214809    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:23:52.214840 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:52.214826    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:23:53.217524 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.217480    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:23:53.217902 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.217551    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:23:53.617397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.617368    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:23:53.640684 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.640656    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config\") pod \"a934850a-3086-4391-b93c-95b96ea4bdb9\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") "
Sep 09 20:23:53.640813 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.640730    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") pod \"a934850a-3086-4391-b93c-95b96ea4bdb9\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") "
Sep 09 20:23:53.640813 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.640756    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zp7x\" (UniqueName: \"kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x\") pod \"a934850a-3086-4391-b93c-95b96ea4bdb9\" (UID: \"a934850a-3086-4391-b93c-95b96ea4bdb9\") "
Sep 09 20:23:53.641019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.640995    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a934850a-3086-4391-b93c-95b96ea4bdb9" (UID: "a934850a-3086-4391-b93c-95b96ea4bdb9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:53.649175 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.649125    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a934850a-3086-4391-b93c-95b96ea4bdb9" (UID: "a934850a-3086-4391-b93c-95b96ea4bdb9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:53.649175 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.649148    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x" (OuterVolumeSpecName: "kube-api-access-5zp7x") pod "a934850a-3086-4391-b93c-95b96ea4bdb9" (UID: "a934850a-3086-4391-b93c-95b96ea4bdb9"). InnerVolumeSpecName "kube-api-access-5zp7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:53.741442 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.741358    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a934850a-3086-4391-b93c-95b96ea4bdb9-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:53.741442 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.741391    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zp7x\" (UniqueName: \"kubernetes.io/projected/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-api-access-5zp7x\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:53.741442 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:53.741405    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a934850a-3086-4391-b93c-95b96ea4bdb9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:54.047151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.047120    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:23:54.112233 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.112196    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:23:54.144619 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.144589    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls\") pod \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") "
Sep 09 20:23:54.144801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.144655    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config\") pod \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") "
Sep 09 20:23:54.144801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.144688    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsvss\" (UniqueName: \"kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss\") pod \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\" (UID: \"8ce218e6-084f-4bb2-adca-e1bd24b173c8\") "
Sep 09 20:23:54.145139 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.145109    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ce218e6-084f-4bb2-adca-e1bd24b173c8" (UID: "8ce218e6-084f-4bb2-adca-e1bd24b173c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:54.147494 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.147462    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss" (OuterVolumeSpecName: "kube-api-access-tsvss") pod "8ce218e6-084f-4bb2-adca-e1bd24b173c8" (UID: "8ce218e6-084f-4bb2-adca-e1bd24b173c8"). InnerVolumeSpecName "kube-api-access-tsvss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:54.147595 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.147540    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ce218e6-084f-4bb2-adca-e1bd24b173c8" (UID: "8ce218e6-084f-4bb2-adca-e1bd24b173c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:54.223365 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.223330    2547 generic.go:356] "Generic (PLEG): container finished" podID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerID="f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4" exitCode=0
Sep 09 20:23:54.223768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.223379    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"
Sep 09 20:23:54.223768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.223412    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerDied","Data":"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"}
Sep 09 20:23:54.223768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.223475    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn" event={"ID":"a934850a-3086-4391-b93c-95b96ea4bdb9","Type":"ContainerDied","Data":"5c171269d9332f8ce4716c3dce8ffbe3ceade93fb8cdcf5568bd6b51dacaa1d1"}
Sep 09 20:23:54.223768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.223516    2547 scope.go:117] "RemoveContainer" containerID="a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"
Sep 09 20:23:54.225318 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.225257    2547 generic.go:356] "Generic (PLEG): container finished" podID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerID="ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c" exitCode=0
Sep 09 20:23:54.225318 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.225282    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerDied","Data":"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"}
Sep 09 20:23:54.225460 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.225330    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"
Sep 09 20:23:54.225460 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.225361    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd" event={"ID":"8ce218e6-084f-4bb2-adca-e1bd24b173c8","Type":"ContainerDied","Data":"f23782f7992fa7905172b2b1d689f4df6a544b3fdf8326c1b93555296a119521"}
Sep 09 20:23:54.232767 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.232750    2547 scope.go:117] "RemoveContainer" containerID="f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"
Sep 09 20:23:54.239942 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.239921    2547 scope.go:117] "RemoveContainer" containerID="a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"
Sep 09 20:23:54.240204 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:54.240186    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661\": container with ID starting with a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661 not found: ID does not exist" containerID="a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"
Sep 09 20:23:54.240249 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.240212    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661"} err="failed to get container status \"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661\": rpc error: code = NotFound desc = could not find container \"a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661\": container with ID starting with a73161af536f3e98c8820ee58e8c2a35ad6bd50bb2131fd7f47c4cda8ebf4661 not found: ID does not exist"
Sep 09 20:23:54.240249 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.240228    2547 scope.go:117] "RemoveContainer" containerID="f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"
Sep 09 20:23:54.240534 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:54.240516    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4\": container with ID starting with f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4 not found: ID does not exist" containerID="f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"
Sep 09 20:23:54.240590 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.240540    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4"} err="failed to get container status \"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4\": rpc error: code = NotFound desc = could not find container \"f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4\": container with ID starting with f4be217a45d6e149c7f673adcc2444e0edfff0cb7cb3144ae9166bed0d3844e4 not found: ID does not exist"
Sep 09 20:23:54.240590 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.240559    2547 scope.go:117] "RemoveContainer" containerID="b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"
Sep 09 20:23:54.245501 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.245478    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:54.245501 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.245503    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvss\" (UniqueName: \"kubernetes.io/projected/8ce218e6-084f-4bb2-adca-e1bd24b173c8-kube-api-access-tsvss\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:54.245660 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.245514    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce218e6-084f-4bb2-adca-e1bd24b173c8-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:23:54.250525 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.250506    2547 scope.go:117] "RemoveContainer" containerID="ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"
Sep 09 20:23:54.251153 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.251131    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:23:54.252636 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.252614    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e029b-predictor-6b5b586b98-847mn"]
Sep 09 20:23:54.258067 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.258051    2547 scope.go:117] "RemoveContainer" containerID="b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"
Sep 09 20:23:54.258357 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:54.258330    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70\": container with ID starting with b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70 not found: ID does not exist" containerID="b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"
Sep 09 20:23:54.258424 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.258365    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70"} err="failed to get container status \"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70\": rpc error: code = NotFound desc = could not find container \"b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70\": container with ID starting with b4f8eed6b770be3e9c1da4b969a7b81cb6b47178ed238d51c8be7706ee8b0f70 not found: ID does not exist"
Sep 09 20:23:54.258424 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.258388    2547 scope.go:117] "RemoveContainer" containerID="ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"
Sep 09 20:23:54.258540 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.258501    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:23:54.258676 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:23:54.258657    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c\": container with ID starting with ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c not found: ID does not exist" containerID="ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"
Sep 09 20:23:54.258733 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.258686    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c"} err="failed to get container status \"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c\": rpc error: code = NotFound desc = could not find container \"ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c\": container with ID starting with ae481d750bb01419a37b2d8fd2b646b1c2aa463b2d7edf6e76cce6b6f0674d8c not found: ID does not exist"
Sep 09 20:23:54.263703 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.263678    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e029b-predictor-6d7b6546c5-qjfgd"]
Sep 09 20:23:54.951664 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.951629    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" path="/var/lib/kubelet/pods/8ce218e6-084f-4bb2-adca-e1bd24b173c8/volumes"
Sep 09 20:23:54.952229 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:54.952208    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" path="/var/lib/kubelet/pods/a934850a-3086-4391-b93c-95b96ea4bdb9/volumes"
Sep 09 20:23:58.222383 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:58.222349    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:23:58.222798 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:58.222602    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:23:58.222843 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:58.222792    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:23:58.223117 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:23:58.223092    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:24:02.102535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:02.102491    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:24:04.113128 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:04.113088    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 09 20:24:08.223331 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:08.223276    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:24:08.223707 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:08.223275    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:24:12.102547 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:12.102504    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 09 20:24:14.113744 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:14.113715    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:24:18.222996 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:18.222950    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:24:18.223380 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:18.223094    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:24:22.103472 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:22.103438    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:24:28.222733 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:28.222694    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 09 20:24:28.223187 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:28.223117    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:24:38.224009 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:38.223975    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:24:38.224536 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:38.224065    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:24:45.523555 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.523522    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:24:45.524019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.523772    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container" containerID="cri-o://651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11" gracePeriod=30
Sep 09 20:24:45.524019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.523823    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kube-rbac-proxy" containerID="cri-o://0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db" gracePeriod=30
Sep 09 20:24:45.566747 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.566711    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:24:45.567013 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.566972    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container" containerID="cri-o://800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73" gracePeriod=30
Sep 09 20:24:45.567132 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.567011    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kube-rbac-proxy" containerID="cri-o://33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d" gracePeriod=30
Sep 09 20:24:45.653455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653422    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:24:45.653751 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653739    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container"
Sep 09 20:24:45.653797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653753    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container"
Sep 09 20:24:45.653797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653764    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653770    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653780    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container"
Sep 09 20:24:45.653797 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653786    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653803    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653808    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653849    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653859    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kube-rbac-proxy"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653866    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce218e6-084f-4bb2-adca-e1bd24b173c8" containerName="kserve-container"
Sep 09 20:24:45.653965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.653870    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="a934850a-3086-4391-b93c-95b96ea4bdb9" containerName="kserve-container"
Sep 09 20:24:45.656975 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.656957    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.658619 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.658597    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a010-predictor-serving-cert\""
Sep 09 20:24:45.658716 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.658602    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a010-kube-rbac-proxy-sar-config\""
Sep 09 20:24:45.666259 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.666234    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:24:45.750150 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.750120    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.750351 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.750180    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.750351 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.750199    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn6c\" (UniqueName: \"kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.752241 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.752214    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:24:45.755613 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.755595    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.757354 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.757332    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a010-predictor-serving-cert\""
Sep 09 20:24:45.757354 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.757343    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a010-kube-rbac-proxy-sar-config\""
Sep 09 20:24:45.765811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.765763    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:24:45.851036 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.850945    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.851036 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851011    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.851230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851039    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn6c\" (UniqueName: \"kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.851230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851097    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.851230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851134    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvkm\" (UniqueName: \"kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.851230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851156    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.851770 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.851744    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.853455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.853437    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.858347 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.858322    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn6c\" (UniqueName: \"kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c\") pod \"success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") " pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:45.951927 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.951893    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.952084 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.951963    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.952084 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.952005    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvkm\" (UniqueName: \"kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.952625 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.952599    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.954276 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.954258    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.959892 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.959873    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvkm\" (UniqueName: \"kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm\") pod \"error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") " pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:45.966698 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:45.966679    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:46.067056 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.067024    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:46.235749 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.235727    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:24:46.237858 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:24:46.237834    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92c8cc4_69f1_42cc_bfb5_39c7ba89de01.slice/crio-2eca38cd868e0a2f4010fc1f369be2f88f7deddf46af646db7ea1c2b69ab25ba WatchSource:0}: Error finding container 2eca38cd868e0a2f4010fc1f369be2f88f7deddf46af646db7ea1c2b69ab25ba: Status 404 returned error can't find the container with id 2eca38cd868e0a2f4010fc1f369be2f88f7deddf46af646db7ea1c2b69ab25ba
Sep 09 20:24:46.337076 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.337051    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:24:46.339411 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:24:46.339388    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9a3567_7b10_43c0_94ac_13a1e6799579.slice/crio-7b153c71de8b15d30d6099641c72740c7a285092967e2924ec1b114b8393c00e WatchSource:0}: Error finding container 7b153c71de8b15d30d6099641c72740c7a285092967e2924ec1b114b8393c00e: Status 404 returned error can't find the container with id 7b153c71de8b15d30d6099641c72740c7a285092967e2924ec1b114b8393c00e
Sep 09 20:24:46.439969 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.439943    2547 generic.go:356] "Generic (PLEG): container finished" podID="58895f95-0127-4cde-bf11-7e427cbcd906" containerID="0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db" exitCode=2
Sep 09 20:24:46.440092 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.440016    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerDied","Data":"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"}
Sep 09 20:24:46.442210 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.442182    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerStarted","Data":"18912d3694b898ba91af3e478b02e8f8b8f58f02d806af4c5e3a271438dcdd25"}
Sep 09 20:24:46.442342 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.442244    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerStarted","Data":"7b153c71de8b15d30d6099641c72740c7a285092967e2924ec1b114b8393c00e"}
Sep 09 20:24:46.444458 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.444432    2547 generic.go:356] "Generic (PLEG): container finished" podID="77ceac87-95c5-40e6-a9b1-d13275442279" containerID="33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d" exitCode=2
Sep 09 20:24:46.444562 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.444505    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerDied","Data":"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"}
Sep 09 20:24:46.446910 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.446886    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerStarted","Data":"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"}
Sep 09 20:24:46.447073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.446942    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerStarted","Data":"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"}
Sep 09 20:24:46.447073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.446975    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerStarted","Data":"2eca38cd868e0a2f4010fc1f369be2f88f7deddf46af646db7ea1c2b69ab25ba"}
Sep 09 20:24:46.447692 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.447675    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:46.447937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.447922    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:46.449085 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.448977    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:24:46.464411 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:46.464352    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podStartSLOduration=1.464337092 podStartE2EDuration="1.464337092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:46.462528439 +0000 UTC m=+1786.033461215" watchObservedRunningTime="2026-09-09 20:24:46.464337092 +0000 UTC m=+1786.035269805"
Sep 09 20:24:47.096705 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.096660    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 09 20:24:47.454809 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.454775    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerStarted","Data":"ecdfde55c407fc884511f4d284cfa363942bfff2fef2f1007d52d67e381bc548"}
Sep 09 20:24:47.455016 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.454921    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:47.455118 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.455090    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:24:47.455175 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.455132    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:47.456276 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.456254    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:24:47.477372 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:47.477261    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podStartSLOduration=2.477245194 podStartE2EDuration="2.477245194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:47.476610122 +0000 UTC m=+1787.047542857" watchObservedRunningTime="2026-09-09 20:24:47.477245194 +0000 UTC m=+1787.048177911"
Sep 09 20:24:48.459320 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:48.459258    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:24:48.459773 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:48.459355    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:24:48.890815 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:48.890792    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:24:48.893683 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:48.893664    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:24:49.081074 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.080997    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config\") pod \"58895f95-0127-4cde-bf11-7e427cbcd906\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") "
Sep 09 20:24:49.081074 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081044    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config\") pod \"77ceac87-95c5-40e6-a9b1-d13275442279\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") "
Sep 09 20:24:49.081245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081163    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7mkl\" (UniqueName: \"kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl\") pod \"58895f95-0127-4cde-bf11-7e427cbcd906\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") "
Sep 09 20:24:49.081245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081199    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls\") pod \"58895f95-0127-4cde-bf11-7e427cbcd906\" (UID: \"58895f95-0127-4cde-bf11-7e427cbcd906\") "
Sep 09 20:24:49.081245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081218    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65gvm\" (UniqueName: \"kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm\") pod \"77ceac87-95c5-40e6-a9b1-d13275442279\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") "
Sep 09 20:24:49.081428 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081410    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls\") pod \"77ceac87-95c5-40e6-a9b1-d13275442279\" (UID: \"77ceac87-95c5-40e6-a9b1-d13275442279\") "
Sep 09 20:24:49.081485 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081422    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58895f95-0127-4cde-bf11-7e427cbcd906" (UID: "58895f95-0127-4cde-bf11-7e427cbcd906"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:49.081485 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081427    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77ceac87-95c5-40e6-a9b1-d13275442279" (UID: "77ceac87-95c5-40e6-a9b1-d13275442279"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:49.081655 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081639    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77ceac87-95c5-40e6-a9b1-d13275442279-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.081710 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.081661    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58895f95-0127-4cde-bf11-7e427cbcd906-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.083301 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.083268    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58895f95-0127-4cde-bf11-7e427cbcd906" (UID: "58895f95-0127-4cde-bf11-7e427cbcd906"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:49.083399 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.083272    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77ceac87-95c5-40e6-a9b1-d13275442279" (UID: "77ceac87-95c5-40e6-a9b1-d13275442279"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:49.083591 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.083562    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl" (OuterVolumeSpecName: "kube-api-access-x7mkl") pod "58895f95-0127-4cde-bf11-7e427cbcd906" (UID: "58895f95-0127-4cde-bf11-7e427cbcd906"). InnerVolumeSpecName "kube-api-access-x7mkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:49.083688 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.083589    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm" (OuterVolumeSpecName: "kube-api-access-65gvm") pod "77ceac87-95c5-40e6-a9b1-d13275442279" (UID: "77ceac87-95c5-40e6-a9b1-d13275442279"). InnerVolumeSpecName "kube-api-access-65gvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:49.182245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.182208    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77ceac87-95c5-40e6-a9b1-d13275442279-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.182245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.182236    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7mkl\" (UniqueName: \"kubernetes.io/projected/58895f95-0127-4cde-bf11-7e427cbcd906-kube-api-access-x7mkl\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.182245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.182245    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58895f95-0127-4cde-bf11-7e427cbcd906-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.182245 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.182254    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65gvm\" (UniqueName: \"kubernetes.io/projected/77ceac87-95c5-40e6-a9b1-d13275442279-kube-api-access-65gvm\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:24:49.465235 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.465205    2547 generic.go:356] "Generic (PLEG): container finished" podID="58895f95-0127-4cde-bf11-7e427cbcd906" containerID="651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11" exitCode=0
Sep 09 20:24:49.465691 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.465261    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"
Sep 09 20:24:49.465691 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.465296    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerDied","Data":"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"}
Sep 09 20:24:49.465691 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.465357    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr" event={"ID":"58895f95-0127-4cde-bf11-7e427cbcd906","Type":"ContainerDied","Data":"8ea99d9f8929bc85ff70d79a57303550453dd47553df57a6612b09750cd0bc96"}
Sep 09 20:24:49.465691 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.465395    2547 scope.go:117] "RemoveContainer" containerID="0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"
Sep 09 20:24:49.467174 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.467156    2547 generic.go:356] "Generic (PLEG): container finished" podID="77ceac87-95c5-40e6-a9b1-d13275442279" containerID="800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73" exitCode=0
Sep 09 20:24:49.467276 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.467210    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerDied","Data":"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"}
Sep 09 20:24:49.467276 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.467230    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"
Sep 09 20:24:49.467276 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.467245    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5" event={"ID":"77ceac87-95c5-40e6-a9b1-d13275442279","Type":"ContainerDied","Data":"cbd812866d07640a05c71ea4a2efafc9b3f81b3e7026c05c6fbfc434a204f951"}
Sep 09 20:24:49.474508 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.474490    2547 scope.go:117] "RemoveContainer" containerID="651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"
Sep 09 20:24:49.481519 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.481501    2547 scope.go:117] "RemoveContainer" containerID="0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"
Sep 09 20:24:49.481784 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:24:49.481765    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db\": container with ID starting with 0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db not found: ID does not exist" containerID="0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"
Sep 09 20:24:49.481852 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.481795    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db"} err="failed to get container status \"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db\": rpc error: code = NotFound desc = could not find container \"0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db\": container with ID starting with 0f51951212cf6040ffbae4cddba1038c0e9102a5e115c351cc5e82026922a5db not found: ID does not exist"
Sep 09 20:24:49.481852 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.481819    2547 scope.go:117] "RemoveContainer" containerID="651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"
Sep 09 20:24:49.482096 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:24:49.482079    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11\": container with ID starting with 651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11 not found: ID does not exist" containerID="651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"
Sep 09 20:24:49.482138 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.482102    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11"} err="failed to get container status \"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11\": rpc error: code = NotFound desc = could not find container \"651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11\": container with ID starting with 651c8d73be533468a08737e5b950dee3e5336a2153db97584425467e3b065e11 not found: ID does not exist"
Sep 09 20:24:49.482138 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.482117    2547 scope.go:117] "RemoveContainer" containerID="33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"
Sep 09 20:24:49.487688 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.487667    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:24:49.489521 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.489508    2547 scope.go:117] "RemoveContainer" containerID="800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"
Sep 09 20:24:49.492323 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.492285    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bf2e8-predictor-c577cf68-h9ggr"]
Sep 09 20:24:49.496370 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.496358    2547 scope.go:117] "RemoveContainer" containerID="33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"
Sep 09 20:24:49.496628 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:24:49.496612    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d\": container with ID starting with 33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d not found: ID does not exist" containerID="33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"
Sep 09 20:24:49.496667 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.496642    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d"} err="failed to get container status \"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d\": rpc error: code = NotFound desc = could not find container \"33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d\": container with ID starting with 33c08faa78f5fce16f64e03ca44749bc4629ac7b6d840c667cc2364cc47c5e3d not found: ID does not exist"
Sep 09 20:24:49.496667 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.496656    2547 scope.go:117] "RemoveContainer" containerID="800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"
Sep 09 20:24:49.496894 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:24:49.496879    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73\": container with ID starting with 800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73 not found: ID does not exist" containerID="800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"
Sep 09 20:24:49.496929 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.496898    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73"} err="failed to get container status \"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73\": rpc error: code = NotFound desc = could not find container \"800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73\": container with ID starting with 800337087a937f0f41b80bb4d94f22728db6764ff2b98c4546a352aeb0693c73 not found: ID does not exist"
Sep 09 20:24:49.503164 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.503145    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:24:49.507974 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:49.507956    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bf2e8-predictor-68cb795cb9-4s2h5"]
Sep 09 20:24:50.950420 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:50.950382    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" path="/var/lib/kubelet/pods/58895f95-0127-4cde-bf11-7e427cbcd906/volumes"
Sep 09 20:24:50.950902 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:50.950780    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" path="/var/lib/kubelet/pods/77ceac87-95c5-40e6-a9b1-d13275442279/volumes"
Sep 09 20:24:53.463518 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:53.463449    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:24:53.463934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:53.463526    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:24:53.463934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:53.463871    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:24:53.463934 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:24:53.463888    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:25:03.465228 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:03.465166    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:25:03.466054 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:03.465779    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:25:13.464075 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:13.464031    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:25:13.464510 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:13.464040    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:25:23.464537 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:23.464498    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:25:23.464943 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:23.464498    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:25:33.464586 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:33.464553    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:25:33.465049 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:25:33.464633    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:33:04.474071 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.474040    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:33:04.476390 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.474251    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container" containerID="cri-o://bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad" gracePeriod=30
Sep 09 20:33:04.476390 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.474304    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kube-rbac-proxy" containerID="cri-o://ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0" gracePeriod=30
Sep 09 20:33:04.623937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.623905    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:33:04.624363 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624337    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kube-rbac-proxy"
Sep 09 20:33:04.624363 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624361    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kube-rbac-proxy"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624374    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624382    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624417    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624426    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624436    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kube-rbac-proxy"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624444    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kube-rbac-proxy"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624518    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624531    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kserve-container"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624542    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="58895f95-0127-4cde-bf11-7e427cbcd906" containerName="kube-rbac-proxy"
Sep 09 20:33:04.624648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.624551    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ceac87-95c5-40e6-a9b1-d13275442279" containerName="kube-rbac-proxy"
Sep 09 20:33:04.627774 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.627751    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.630423 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.630400    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-97f7b-kube-rbac-proxy-sar-config\""
Sep 09 20:33:04.631356 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.631337    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-97f7b-predictor-serving-cert\""
Sep 09 20:33:04.639059 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.639033    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:33:04.657737 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.657702    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:33:04.657968 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.657947    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container" containerID="cri-o://52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148" gracePeriod=30
Sep 09 20:33:04.658072 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.658037    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kube-rbac-proxy" containerID="cri-o://2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a" gracePeriod=30
Sep 09 20:33:04.712969 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.712937    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:33:04.716128 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.716112    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.718272 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.718245    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-97f7b-predictor-serving-cert\""
Sep 09 20:33:04.718395 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.718299    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-97f7b-kube-rbac-proxy-sar-config\""
Sep 09 20:33:04.726101 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.726049    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:33:04.739201 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.739177    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.739331 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.739225    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhl46\" (UniqueName: \"kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.739331 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.739276    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.840423 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840389    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhl46\" (UniqueName: \"kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.840597 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840430    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.840597 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840451    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.840597 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840476    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdvz\" (UniqueName: \"kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.840597 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840559    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.840801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.840607    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.841164 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.841148    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.842880 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.842851    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.852318 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.849103    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhl46\" (UniqueName: \"kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46\") pod \"success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") " pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.939220 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.939183    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:04.941648 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.941615    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.941778 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.941709    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.941778 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.941752    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdvz\" (UniqueName: \"kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.942402 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.942375    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.944356 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.944331    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:04.957346 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:04.957320    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdvz\" (UniqueName: \"kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz\") pod \"error-404-isvc-97f7b-predictor-5c667894df-2bfzn\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") " pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:05.026747 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.026711    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:05.211873 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.211805    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:33:05.213961 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:33:05.213936    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeb1746_bbcc_463f_a3f3_dd7d089e0cb4.slice/crio-05d7cd8d795982f31f4b69d7d6114edffaacfc23b18daa055ec2fafe5eaaa795 WatchSource:0}: Error finding container 05d7cd8d795982f31f4b69d7d6114edffaacfc23b18daa055ec2fafe5eaaa795: Status 404 returned error can't find the container with id 05d7cd8d795982f31f4b69d7d6114edffaacfc23b18daa055ec2fafe5eaaa795
Sep 09 20:33:05.302247 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.302225    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:33:05.303887 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:33:05.303865    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a2f6b7_48cb_4a1e_a91c_3fc43aa6aa16.slice/crio-2df6bac6c7106cab64e7474f0126c48b9f85fd70803caabd632da67818ebf0f8 WatchSource:0}: Error finding container 2df6bac6c7106cab64e7474f0126c48b9f85fd70803caabd632da67818ebf0f8: Status 404 returned error can't find the container with id 2df6bac6c7106cab64e7474f0126c48b9f85fd70803caabd632da67818ebf0f8
Sep 09 20:33:05.421562 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.421527    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerStarted","Data":"01471d604ca413900db571eb95e0de386e01470510c47def7444524b227597f9"}
Sep 09 20:33:05.421683 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.421596    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerStarted","Data":"992411527c349839e5a5cdac29fa68ca4c07e74a9afaf0938f8b60207a738f65"}
Sep 09 20:33:05.421683 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.421629    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:05.421683 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.421658    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerStarted","Data":"05d7cd8d795982f31f4b69d7d6114edffaacfc23b18daa055ec2fafe5eaaa795"}
Sep 09 20:33:05.423255 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.423227    2547 generic.go:356] "Generic (PLEG): container finished" podID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerID="ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0" exitCode=2
Sep 09 20:33:05.423365 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.423309    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerDied","Data":"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"}
Sep 09 20:33:05.425105 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.425083    2547 generic.go:356] "Generic (PLEG): container finished" podID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerID="2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a" exitCode=2
Sep 09 20:33:05.425248 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.425146    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerDied","Data":"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"}
Sep 09 20:33:05.426796 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.426776    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerStarted","Data":"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"}
Sep 09 20:33:05.426874 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.426824    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerStarted","Data":"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"}
Sep 09 20:33:05.426874 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.426855    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:05.426874 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.426873    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerStarted","Data":"2df6bac6c7106cab64e7474f0126c48b9f85fd70803caabd632da67818ebf0f8"}
Sep 09 20:33:05.426989 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.426901    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:05.428418 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.428393    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:05.477203 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.477149    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podStartSLOduration=1.47713062 podStartE2EDuration="1.47713062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:05.441281033 +0000 UTC m=+2285.012213769" watchObservedRunningTime="2026-09-09 20:33:05.47713062 +0000 UTC m=+2285.048063335"
Sep 09 20:33:05.477589 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:05.477429    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podStartSLOduration=1.477421972 podStartE2EDuration="1.477421972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:05.475769202 +0000 UTC m=+2285.046701919" watchObservedRunningTime="2026-09-09 20:33:05.477421972 +0000 UTC m=+2285.048354721"
Sep 09 20:33:06.431617 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:06.431588    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:06.431810 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:06.431613    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:06.432954 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:06.432928    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:07.435855 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.435815    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:07.712406 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.712378    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:33:07.804138 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.804115    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:33:07.864509 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.864485    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls\") pod \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") "
Sep 09 20:33:07.864660 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.864519    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config\") pod \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") "
Sep 09 20:33:07.864660 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.864571    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9dzn\" (UniqueName: \"kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn\") pod \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\" (UID: \"989b24c7-2c45-46e5-82aa-556c7ebcdb0e\") "
Sep 09 20:33:07.864921 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.864900    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "989b24c7-2c45-46e5-82aa-556c7ebcdb0e" (UID: "989b24c7-2c45-46e5-82aa-556c7ebcdb0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:07.866592 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.866568    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn" (OuterVolumeSpecName: "kube-api-access-z9dzn") pod "989b24c7-2c45-46e5-82aa-556c7ebcdb0e" (UID: "989b24c7-2c45-46e5-82aa-556c7ebcdb0e"). InnerVolumeSpecName "kube-api-access-z9dzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:07.866675 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.866605    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "989b24c7-2c45-46e5-82aa-556c7ebcdb0e" (UID: "989b24c7-2c45-46e5-82aa-556c7ebcdb0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:07.965856 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.965780    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") pod \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") "
Sep 09 20:33:07.965856 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.965840    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zxhq\" (UniqueName: \"kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq\") pod \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") "
Sep 09 20:33:07.966019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.965872    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config\") pod \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\" (UID: \"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20\") "
Sep 09 20:33:07.966019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.965979    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:07.966019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.965991    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:07.966019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.966000    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9dzn\" (UniqueName: \"kubernetes.io/projected/989b24c7-2c45-46e5-82aa-556c7ebcdb0e-kube-api-access-z9dzn\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:07.966285 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.966258    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" (UID: "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:07.967889 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.967866    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq" (OuterVolumeSpecName: "kube-api-access-5zxhq") pod "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" (UID: "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20"). InnerVolumeSpecName "kube-api-access-5zxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:07.967977 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:07.967871    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" (UID: "2f8c5e33-99fa-4e19-9286-eb3f5ec93e20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:08.066364 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.066333    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:08.066364 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.066360    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:08.066521 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.066373    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxhq\" (UniqueName: \"kubernetes.io/projected/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20-kube-api-access-5zxhq\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:33:08.441101 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.441070    2547 generic.go:356] "Generic (PLEG): container finished" podID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerID="bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad" exitCode=0
Sep 09 20:33:08.441530 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.441117    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"
Sep 09 20:33:08.441530 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.441150    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerDied","Data":"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"}
Sep 09 20:33:08.441530 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.441209    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4" event={"ID":"989b24c7-2c45-46e5-82aa-556c7ebcdb0e","Type":"ContainerDied","Data":"c9a594c0af3b3fa10b6d555c049f5564144b3d9039d0aeb3475bb5da45a1b01a"}
Sep 09 20:33:08.441530 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.441241    2547 scope.go:117] "RemoveContainer" containerID="ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"
Sep 09 20:33:08.443011 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.442985    2547 generic.go:356] "Generic (PLEG): container finished" podID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerID="52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148" exitCode=0
Sep 09 20:33:08.443139 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.443054    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"
Sep 09 20:33:08.443139 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.443069    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerDied","Data":"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"}
Sep 09 20:33:08.443139 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.443136    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7" event={"ID":"2f8c5e33-99fa-4e19-9286-eb3f5ec93e20","Type":"ContainerDied","Data":"b61c9f6e51d7a871e6268542ac4ebb46d188429b215f9a7a36b00ca719f91f3e"}
Sep 09 20:33:08.451113 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.451096    2547 scope.go:117] "RemoveContainer" containerID="bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"
Sep 09 20:33:08.458670 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.458654    2547 scope.go:117] "RemoveContainer" containerID="ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"
Sep 09 20:33:08.458904 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:33:08.458885    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0\": container with ID starting with ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0 not found: ID does not exist" containerID="ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"
Sep 09 20:33:08.458962 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.458912    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0"} err="failed to get container status \"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0\": rpc error: code = NotFound desc = could not find container \"ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0\": container with ID starting with ce200c0bda9cdc486b6364047128b432c043df9146b6315ac31cbcc16127f0d0 not found: ID does not exist"
Sep 09 20:33:08.458962 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.458928    2547 scope.go:117] "RemoveContainer" containerID="bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"
Sep 09 20:33:08.459209 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:33:08.459184    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad\": container with ID starting with bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad not found: ID does not exist" containerID="bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"
Sep 09 20:33:08.459324 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.459212    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad"} err="failed to get container status \"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad\": rpc error: code = NotFound desc = could not find container \"bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad\": container with ID starting with bcf8d8d58eeba0476b065d5c37f99999a2d0b6f49089732f84f8d00a9fb568ad not found: ID does not exist"
Sep 09 20:33:08.459324 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.459226    2547 scope.go:117] "RemoveContainer" containerID="2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"
Sep 09 20:33:08.465498 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.465448    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:33:08.468004 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.467967    2547 scope.go:117] "RemoveContainer" containerID="52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"
Sep 09 20:33:08.469740 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.469719    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3c295-predictor-c97b4b754-44xk4"]
Sep 09 20:33:08.475221 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.475205    2547 scope.go:117] "RemoveContainer" containerID="2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"
Sep 09 20:33:08.475483 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:33:08.475465    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a\": container with ID starting with 2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a not found: ID does not exist" containerID="2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"
Sep 09 20:33:08.475541 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.475488    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a"} err="failed to get container status \"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a\": rpc error: code = NotFound desc = could not find container \"2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a\": container with ID starting with 2ac41e0e42af1d968665c84fb61ae8ab3d38abef5e5681518c18dcfd53a39b4a not found: ID does not exist"
Sep 09 20:33:08.475541 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.475503    2547 scope.go:117] "RemoveContainer" containerID="52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"
Sep 09 20:33:08.475767 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:33:08.475751    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148\": container with ID starting with 52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148 not found: ID does not exist" containerID="52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"
Sep 09 20:33:08.475811 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.475770    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148"} err="failed to get container status \"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148\": rpc error: code = NotFound desc = could not find container \"52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148\": container with ID starting with 52d2b786211c25b65f6b74967586c6764614b266f3136e6bd5f3308fdca7a148 not found: ID does not exist"
Sep 09 20:33:08.483188 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.483169    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:33:08.490490 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.490472    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3c295-predictor-6f998d6587-44pj7"]
Sep 09 20:33:08.947712 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.947671    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" path="/var/lib/kubelet/pods/2f8c5e33-99fa-4e19-9286-eb3f5ec93e20/volumes"
Sep 09 20:33:08.948188 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:08.948167    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" path="/var/lib/kubelet/pods/989b24c7-2c45-46e5-82aa-556c7ebcdb0e/volumes"
Sep 09 20:33:11.435611 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:11.435579    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:11.436018 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:11.435993    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:12.440121 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:12.440090    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:33:12.440728 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:12.440701    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:21.436688 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:21.436645    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:22.441170 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:22.441127    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:31.436696 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:31.436659    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:32.440701 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:32.440663    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:41.436695 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:41.436629    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:33:42.441189 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:42.441151    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:33:51.437259 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:51.437230    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:33:52.442046 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:33:52.442017    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:34:00.384767 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.384661    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:34:00.385328 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.384948    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" containerID="cri-o://052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1" gracePeriod=30
Sep 09 20:34:00.385328 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.385039    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kube-rbac-proxy" containerID="cri-o://2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242" gracePeriod=30
Sep 09 20:34:00.420767 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.420736    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:34:00.421029 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.420999    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" containerID="cri-o://18912d3694b898ba91af3e478b02e8f8b8f58f02d806af4c5e3a271438dcdd25" gracePeriod=30
Sep 09 20:34:00.421170 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.421038    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kube-rbac-proxy" containerID="cri-o://ecdfde55c407fc884511f4d284cfa363942bfff2fef2f1007d52d67e381bc548" gracePeriod=30
Sep 09 20:34:00.530230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530201    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:34:00.530554 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530539    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kube-rbac-proxy"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530555    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kube-rbac-proxy"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530573    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530578    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530586    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530592    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container"
Sep 09 20:34:00.530603 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530602    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kube-rbac-proxy"
Sep 09 20:34:00.530782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530608    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kube-rbac-proxy"
Sep 09 20:34:00.530782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530653    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kserve-container"
Sep 09 20:34:00.530782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530659    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kserve-container"
Sep 09 20:34:00.530782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530666    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="989b24c7-2c45-46e5-82aa-556c7ebcdb0e" containerName="kube-rbac-proxy"
Sep 09 20:34:00.530782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.530671    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8c5e33-99fa-4e19-9286-eb3f5ec93e20" containerName="kube-rbac-proxy"
Sep 09 20:34:00.533531 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.533514    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.535415 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.535394    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-54686-predictor-serving-cert\""
Sep 09 20:34:00.535535 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.535410    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-54686-kube-rbac-proxy-sar-config\""
Sep 09 20:34:00.545545 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.545519    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:34:00.620098 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.620071    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:34:00.623327 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.623310    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.625207 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.625192    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-54686-predictor-serving-cert\""
Sep 09 20:34:00.625322 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.625241    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-54686-kube-rbac-proxy-sar-config\""
Sep 09 20:34:00.633225 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.633203    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:34:00.663249 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.663188    2547 generic.go:356] "Generic (PLEG): container finished" podID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerID="ecdfde55c407fc884511f4d284cfa363942bfff2fef2f1007d52d67e381bc548" exitCode=2
Sep 09 20:34:00.663382 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.663264    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerDied","Data":"ecdfde55c407fc884511f4d284cfa363942bfff2fef2f1007d52d67e381bc548"}
Sep 09 20:34:00.664911 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.664889    2547 generic.go:356] "Generic (PLEG): container finished" podID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerID="2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242" exitCode=2
Sep 09 20:34:00.665009 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.664939    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerDied","Data":"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"}
Sep 09 20:34:00.674498 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.674474    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.674589 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.674524    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.674640 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.674612    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5c6b\" (UniqueName: \"kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.775873 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.775836    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.776073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.775886    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.776073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.776020    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c6b\" (UniqueName: \"kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.776073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.776068    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.776231 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.776140    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.776231 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.776166    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqh4\" (UniqueName: \"kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.776722 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.776698    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.778322 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.778285    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.783054 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.783034    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c6b\" (UniqueName: \"kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b\") pod \"success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") " pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.845456 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.845430    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:00.877330 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.877269    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.877442 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.877348    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqh4\" (UniqueName: \"kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.877491 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.877476    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.879344 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.879322    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-54686-kube-rbac-proxy-sar-config\""
Sep 09 20:34:00.879471 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.879371    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-54686-predictor-serving-cert\""
Sep 09 20:34:00.885528 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.885507    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqh4\" (UniqueName: \"kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.888713 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.888692    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.890234 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.890215    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls\") pod \"error-404-isvc-54686-predictor-5478745645-rdwpn\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") " pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:00.933938 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:00.933874    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:01.117039 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.117007    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:34:01.119749 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:34:01.119728    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0879e79_177e_48a8_8992_e60a36b9898d.slice/crio-24bbf0e5f9755f438263641b8199927a6f3d0f09b3fb995abba4a5e77300b6ca WatchSource:0}: Error finding container 24bbf0e5f9755f438263641b8199927a6f3d0f09b3fb995abba4a5e77300b6ca: Status 404 returned error can't find the container with id 24bbf0e5f9755f438263641b8199927a6f3d0f09b3fb995abba4a5e77300b6ca
Sep 09 20:34:01.215027 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.214999    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:34:01.218443 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:34:01.218421    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7150978e_3939_47c3_bc39_626cd5d3fb92.slice/crio-d84a834fb921f176617b21d8590534f2df1ab605470613b4f830bbc75b17a387 WatchSource:0}: Error finding container d84a834fb921f176617b21d8590534f2df1ab605470613b4f830bbc75b17a387: Status 404 returned error can't find the container with id d84a834fb921f176617b21d8590534f2df1ab605470613b4f830bbc75b17a387
Sep 09 20:34:01.670882 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.670844    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerStarted","Data":"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"}
Sep 09 20:34:01.671369 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.671013    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:01.671369 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.671083    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerStarted","Data":"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"}
Sep 09 20:34:01.671369 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.671120    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerStarted","Data":"d84a834fb921f176617b21d8590534f2df1ab605470613b4f830bbc75b17a387"}
Sep 09 20:34:01.671369 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.671153    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:01.672557 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.672529    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:01.672861 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.672839    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerStarted","Data":"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"}
Sep 09 20:34:01.672936 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.672893    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerStarted","Data":"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"}
Sep 09 20:34:01.673005 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.672970    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:01.673005 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.673002    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerStarted","Data":"24bbf0e5f9755f438263641b8199927a6f3d0f09b3fb995abba4a5e77300b6ca"}
Sep 09 20:34:01.673103 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.673033    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:01.673895 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.673873    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:01.687736 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.687694    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podStartSLOduration=1.687680591 podStartE2EDuration="1.687680591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:01.686407469 +0000 UTC m=+2341.257340182" watchObservedRunningTime="2026-09-09 20:34:01.687680591 +0000 UTC m=+2341.258613307"
Sep 09 20:34:01.703318 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:01.703259    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podStartSLOduration=1.70324572 podStartE2EDuration="1.70324572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:01.701546481 +0000 UTC m=+2341.272479196" watchObservedRunningTime="2026-09-09 20:34:01.70324572 +0000 UTC m=+2341.274178434"
Sep 09 20:34:02.678818 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:02.678780    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:02.679230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:02.678850    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:03.460128 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.460089    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 09 20:34:03.460280 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.460092    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 09 20:34:03.464403 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.464382    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:34:03.464470 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.464388    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:34:03.685063 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.685030    2547 generic.go:356] "Generic (PLEG): container finished" podID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerID="18912d3694b898ba91af3e478b02e8f8b8f58f02d806af4c5e3a271438dcdd25" exitCode=0
Sep 09 20:34:03.685541 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.685096    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerDied","Data":"18912d3694b898ba91af3e478b02e8f8b8f58f02d806af4c5e3a271438dcdd25"}
Sep 09 20:34:03.776818 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.776794    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:34:03.905515 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.905477    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls\") pod \"af9a3567-7b10-43c0-94ac-13a1e6799579\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") "
Sep 09 20:34:03.905643 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.905535    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config\") pod \"af9a3567-7b10-43c0-94ac-13a1e6799579\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") "
Sep 09 20:34:03.905643 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.905606    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrvkm\" (UniqueName: \"kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm\") pod \"af9a3567-7b10-43c0-94ac-13a1e6799579\" (UID: \"af9a3567-7b10-43c0-94ac-13a1e6799579\") "
Sep 09 20:34:03.905940 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.905907    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af9a3567-7b10-43c0-94ac-13a1e6799579" (UID: "af9a3567-7b10-43c0-94ac-13a1e6799579"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:03.907468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.907444    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af9a3567-7b10-43c0-94ac-13a1e6799579" (UID: "af9a3567-7b10-43c0-94ac-13a1e6799579"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:03.907579 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:03.907546    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm" (OuterVolumeSpecName: "kube-api-access-qrvkm") pod "af9a3567-7b10-43c0-94ac-13a1e6799579" (UID: "af9a3567-7b10-43c0-94ac-13a1e6799579"). InnerVolumeSpecName "kube-api-access-qrvkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:04.006397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.006373    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvkm\" (UniqueName: \"kubernetes.io/projected/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-api-access-qrvkm\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.006397 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.006396    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af9a3567-7b10-43c0-94ac-13a1e6799579-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.006539 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.006409    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af9a3567-7b10-43c0-94ac-13a1e6799579-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.011950 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.011894    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:34:04.106693 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.106666    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsn6c\" (UniqueName: \"kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c\") pod \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") "
Sep 09 20:34:04.106829 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.106706    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls\") pod \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") "
Sep 09 20:34:04.106829 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.106753    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config\") pod \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\" (UID: \"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01\") "
Sep 09 20:34:04.107218 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.107191    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" (UID: "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:04.108885 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.108861    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" (UID: "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:04.108990 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.108900    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c" (OuterVolumeSpecName: "kube-api-access-fsn6c") pod "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" (UID: "e92c8cc4-69f1-42cc-bfb5-39c7ba89de01"). InnerVolumeSpecName "kube-api-access-fsn6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:04.207573 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.207541    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn6c\" (UniqueName: \"kubernetes.io/projected/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-api-access-fsn6c\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.207573 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.207569    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.207573 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.207580    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:04.690336 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.690303    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"
Sep 09 20:34:04.690792 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.690305    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw" event={"ID":"af9a3567-7b10-43c0-94ac-13a1e6799579","Type":"ContainerDied","Data":"7b153c71de8b15d30d6099641c72740c7a285092967e2924ec1b114b8393c00e"}
Sep 09 20:34:04.690792 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.690449    2547 scope.go:117] "RemoveContainer" containerID="ecdfde55c407fc884511f4d284cfa363942bfff2fef2f1007d52d67e381bc548"
Sep 09 20:34:04.692188 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.692164    2547 generic.go:356] "Generic (PLEG): container finished" podID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerID="052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1" exitCode=0
Sep 09 20:34:04.692341 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.692211    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"
Sep 09 20:34:04.692341 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.692253    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerDied","Data":"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"}
Sep 09 20:34:04.692468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.692353    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2" event={"ID":"e92c8cc4-69f1-42cc-bfb5-39c7ba89de01","Type":"ContainerDied","Data":"2eca38cd868e0a2f4010fc1f369be2f88f7deddf46af646db7ea1c2b69ab25ba"}
Sep 09 20:34:04.699552 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.699530    2547 scope.go:117] "RemoveContainer" containerID="18912d3694b898ba91af3e478b02e8f8b8f58f02d806af4c5e3a271438dcdd25"
Sep 09 20:34:04.707089 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.707072    2547 scope.go:117] "RemoveContainer" containerID="2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"
Sep 09 20:34:04.711171 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.711152    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:34:04.713314 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.713274    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a010-predictor-65b6cbcbcc-gt8mw"]
Sep 09 20:34:04.714404 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.714387    2547 scope.go:117] "RemoveContainer" containerID="052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"
Sep 09 20:34:04.721193 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.721165    2547 scope.go:117] "RemoveContainer" containerID="2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"
Sep 09 20:34:04.721506 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:04.721480    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242\": container with ID starting with 2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242 not found: ID does not exist" containerID="2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"
Sep 09 20:34:04.721584 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.721515    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242"} err="failed to get container status \"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242\": rpc error: code = NotFound desc = could not find container \"2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242\": container with ID starting with 2e1095bffa5af8fa92bfae3e9cc9cac13bc1cbfc5f25686a7e22801090d0b242 not found: ID does not exist"
Sep 09 20:34:04.721584 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.721534    2547 scope.go:117] "RemoveContainer" containerID="052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"
Sep 09 20:34:04.721875 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:04.721855    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1\": container with ID starting with 052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1 not found: ID does not exist" containerID="052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"
Sep 09 20:34:04.722002 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.721880    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1"} err="failed to get container status \"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1\": rpc error: code = NotFound desc = could not find container \"052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1\": container with ID starting with 052025a20524f92d54bbaee79c95fcb9b0920cd68c396be9332fd86d49a273e1 not found: ID does not exist"
Sep 09 20:34:04.723235 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.723216    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:34:04.729546 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.729522    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a010-predictor-7d55bcfc76-5ggw2"]
Sep 09 20:34:04.948383 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.948350    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" path="/var/lib/kubelet/pods/af9a3567-7b10-43c0-94ac-13a1e6799579/volumes"
Sep 09 20:34:04.948765 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:04.948753    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" path="/var/lib/kubelet/pods/e92c8cc4-69f1-42cc-bfb5-39c7ba89de01/volumes"
Sep 09 20:34:07.682709 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:07.682672    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:34:07.683194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:07.683063    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:07.683194 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:07.683103    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:07.683476 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:07.683455    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:14.802016 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.801984    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:34:14.802455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.802214    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container" containerID="cri-o://992411527c349839e5a5cdac29fa68ca4c07e74a9afaf0938f8b60207a738f65" gracePeriod=30
Sep 09 20:34:14.802455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.802244    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kube-rbac-proxy" containerID="cri-o://01471d604ca413900db571eb95e0de386e01470510c47def7444524b227597f9" gracePeriod=30
Sep 09 20:34:14.847474 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.847443    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:34:14.847695 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.847668    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container" containerID="cri-o://35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d" gracePeriod=30
Sep 09 20:34:14.848091 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.847949    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kube-rbac-proxy" containerID="cri-o://fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4" gracePeriod=30
Sep 09 20:34:14.948078 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948045    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:34:14.948424 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948408    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948424 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948426    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948438    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948446    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948470    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948478    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948490    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948497    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948849 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948575    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kserve-container"
Sep 09 20:34:14.948849 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948587    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92c8cc4-69f1-42cc-bfb5-39c7ba89de01" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948849 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948601    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kube-rbac-proxy"
Sep 09 20:34:14.948849 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.948612    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9a3567-7b10-43c0-94ac-13a1e6799579" containerName="kserve-container"
Sep 09 20:34:14.951499 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.951473    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:14.953347 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.953323    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b9ccb-predictor-serving-cert\""
Sep 09 20:34:14.953347 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.953323    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b9ccb-kube-rbac-proxy-sar-config\""
Sep 09 20:34:14.961801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:14.961777    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:34:15.045707 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.045669    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:34:15.049040 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.049013    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.050850 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.050830    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b9ccb-predictor-serving-cert\""
Sep 09 20:34:15.051018 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.050830    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b9ccb-kube-rbac-proxy-sar-config\""
Sep 09 20:34:15.058746 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.058723    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:34:15.091675 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.091638    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.091871 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.091691    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.091871 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.091767    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wsf\" (UniqueName: \"kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.192704 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192671    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrp5\" (UniqueName: \"kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.192858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192713    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.192858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192751    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.192858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192788    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wsf\" (UniqueName: \"kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.192858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192810    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.192858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.192840    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.193467 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.193443    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.195318 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.195277    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.199743 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.199721    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wsf\" (UniqueName: \"kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf\") pod \"success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") " pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.261536 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.261506    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.294068 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.294035    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.294210 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.294084    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrp5\" (UniqueName: \"kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.294210 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.294120    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.294210 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:15.294189    2547 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-serving-cert: secret "error-404-isvc-b9ccb-predictor-serving-cert" not found
Sep 09 20:34:15.294403 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:15.294266    2547 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls podName:7ae33901-ff0d-4f8a-b058-c56f41c2f828 nodeName:}" failed. No retries permitted until 2026-09-09 20:34:15.794245431 +0000 UTC m=+2355.365178137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls") pod "error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" (UID: "7ae33901-ff0d-4f8a-b058-c56f41c2f828") : secret "error-404-isvc-b9ccb-predictor-serving-cert" not found
Sep 09 20:34:15.294780 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.294757    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.301851 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.301830    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrp5\" (UniqueName: \"kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.528953 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.528921    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:34:15.531753 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:34:15.531723    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06caf308_6682_41dd_a365_1c0a17cc0e6e.slice/crio-c0f96c6ec3d44277d0e934f0d0ab91564ece53bfd24ce9e9470c350b3112795a WatchSource:0}: Error finding container c0f96c6ec3d44277d0e934f0d0ab91564ece53bfd24ce9e9470c350b3112795a: Status 404 returned error can't find the container with id c0f96c6ec3d44277d0e934f0d0ab91564ece53bfd24ce9e9470c350b3112795a
Sep 09 20:34:15.743833 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.743789    2547 generic.go:356] "Generic (PLEG): container finished" podID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerID="fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4" exitCode=2
Sep 09 20:34:15.744019 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.743899    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerDied","Data":"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"}
Sep 09 20:34:15.747700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.746631    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerStarted","Data":"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"}
Sep 09 20:34:15.747700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.746700    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerStarted","Data":"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"}
Sep 09 20:34:15.747700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.746734    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerStarted","Data":"c0f96c6ec3d44277d0e934f0d0ab91564ece53bfd24ce9e9470c350b3112795a"}
Sep 09 20:34:15.747700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.747596    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.747700 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.747647    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:15.748937 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.748805    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:15.750683 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.750652    2547 generic.go:356] "Generic (PLEG): container finished" podID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerID="01471d604ca413900db571eb95e0de386e01470510c47def7444524b227597f9" exitCode=2
Sep 09 20:34:15.750814 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.750712    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerDied","Data":"01471d604ca413900db571eb95e0de386e01470510c47def7444524b227597f9"}
Sep 09 20:34:15.764550 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.764505    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podStartSLOduration=1.764489725 podStartE2EDuration="1.764489725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:15.764053631 +0000 UTC m=+2355.334986345" watchObservedRunningTime="2026-09-09 20:34:15.764489725 +0000 UTC m=+2355.335422438"
Sep 09 20:34:15.797699 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.797674    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.800101 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.800081    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") pod \"error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") " pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:15.959562 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:15.959528    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:16.227593 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.227518    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:34:16.230466 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:34:16.230442    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae33901_ff0d_4f8a_b058_c56f41c2f828.slice/crio-26093c63e942db6d09d4dbd6097c384b63b2a095b486c77ebca6af3e1e04fb8a WatchSource:0}: Error finding container 26093c63e942db6d09d4dbd6097c384b63b2a095b486c77ebca6af3e1e04fb8a: Status 404 returned error can't find the container with id 26093c63e942db6d09d4dbd6097c384b63b2a095b486c77ebca6af3e1e04fb8a
Sep 09 20:34:16.432582 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.432547    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 09 20:34:16.758872 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.758837    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerStarted","Data":"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"}
Sep 09 20:34:16.759078 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.758905    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerStarted","Data":"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"}
Sep 09 20:34:16.759078 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.758936    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerStarted","Data":"26093c63e942db6d09d4dbd6097c384b63b2a095b486c77ebca6af3e1e04fb8a"}
Sep 09 20:34:16.759208 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.759188    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:16.759268 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.759250    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:16.759417 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.759390    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:16.760468 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.760443    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:16.775634 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:16.775582    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podStartSLOduration=1.775562576 podStartE2EDuration="1.775562576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:16.774792723 +0000 UTC m=+2356.345725447" watchObservedRunningTime="2026-09-09 20:34:16.775562576 +0000 UTC m=+2356.346495293"
Sep 09 20:34:17.436523 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:17.436481    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 09 20:34:17.683102 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:17.683065    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:17.683504 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:17.683481    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:17.764428 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:17.764335    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:17.764428 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:17.764367    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:18.096381 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.096356    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:34:18.219250 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.219217    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls\") pod \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") "
Sep 09 20:34:18.219444 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.219266    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsdvz\" (UniqueName: \"kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz\") pod \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") "
Sep 09 20:34:18.219444 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.219354    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config\") pod \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\" (UID: \"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16\") "
Sep 09 20:34:18.219731 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.219705    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" (UID: "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:18.221241 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.221221    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" (UID: "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:18.221368 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.221353    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz" (OuterVolumeSpecName: "kube-api-access-vsdvz") pod "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" (UID: "f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16"). InnerVolumeSpecName "kube-api-access-vsdvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:18.320440 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.320361    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:18.320440 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.320387    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:18.320440 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.320403    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsdvz\" (UniqueName: \"kubernetes.io/projected/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16-kube-api-access-vsdvz\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:18.771829 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.771800    2547 generic.go:356] "Generic (PLEG): container finished" podID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerID="35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d" exitCode=0
Sep 09 20:34:18.772248 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.771848    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"
Sep 09 20:34:18.772248 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.771876    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerDied","Data":"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"}
Sep 09 20:34:18.772248 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.771939    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn" event={"ID":"f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16","Type":"ContainerDied","Data":"2df6bac6c7106cab64e7474f0126c48b9f85fd70803caabd632da67818ebf0f8"}
Sep 09 20:34:18.772248 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.771978    2547 scope.go:117] "RemoveContainer" containerID="fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"
Sep 09 20:34:18.774433 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.774395    2547 generic.go:356] "Generic (PLEG): container finished" podID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerID="992411527c349839e5a5cdac29fa68ca4c07e74a9afaf0938f8b60207a738f65" exitCode=0
Sep 09 20:34:18.774562 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.774482    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerDied","Data":"992411527c349839e5a5cdac29fa68ca4c07e74a9afaf0938f8b60207a738f65"}
Sep 09 20:34:18.781841 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.781826    2547 scope.go:117] "RemoveContainer" containerID="35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"
Sep 09 20:34:18.790299 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.790265    2547 scope.go:117] "RemoveContainer" containerID="fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"
Sep 09 20:34:18.790576 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:18.790557    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4\": container with ID starting with fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4 not found: ID does not exist" containerID="fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"
Sep 09 20:34:18.790638 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.790583    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4"} err="failed to get container status \"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4\": rpc error: code = NotFound desc = could not find container \"fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4\": container with ID starting with fec3d4aee3e076623482c25c5cbad6f990d0d9c2104fc8fdd56b7002e13dd9a4 not found: ID does not exist"
Sep 09 20:34:18.790638 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.790600    2547 scope.go:117] "RemoveContainer" containerID="35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"
Sep 09 20:34:18.790875 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:34:18.790856    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d\": container with ID starting with 35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d not found: ID does not exist" containerID="35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"
Sep 09 20:34:18.790933 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.790884    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d"} err="failed to get container status \"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d\": rpc error: code = NotFound desc = could not find container \"35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d\": container with ID starting with 35ae80cf4e73f577fe6d56f45bea1795111079e1854203a699edd9678a17371d not found: ID does not exist"
Sep 09 20:34:18.793013 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.792994    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:34:18.796950 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.796930    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-97f7b-predictor-5c667894df-2bfzn"]
Sep 09 20:34:18.837081 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.837060    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:34:18.947941 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:18.947855    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" path="/var/lib/kubelet/pods/f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16/volumes"
Sep 09 20:34:19.024774 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.024747    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhl46\" (UniqueName: \"kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46\") pod \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") "
Sep 09 20:34:19.024892 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.024801    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls\") pod \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") "
Sep 09 20:34:19.024892 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.024824    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config\") pod \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\" (UID: \"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4\") "
Sep 09 20:34:19.025203 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.025180    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" (UID: "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:19.026869 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.026832    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" (UID: "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:19.026968 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.026898    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46" (OuterVolumeSpecName: "kube-api-access-mhl46") pod "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" (UID: "ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4"). InnerVolumeSpecName "kube-api-access-mhl46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:19.125999 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.125957    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhl46\" (UniqueName: \"kubernetes.io/projected/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-api-access-mhl46\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:19.125999 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.125994    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:19.125999 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.126006    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:34:19.780139 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.780105    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"
Sep 09 20:34:19.780642 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.780124    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb" event={"ID":"ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4","Type":"ContainerDied","Data":"05d7cd8d795982f31f4b69d7d6114edffaacfc23b18daa055ec2fafe5eaaa795"}
Sep 09 20:34:19.780642 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.780214    2547 scope.go:117] "RemoveContainer" containerID="01471d604ca413900db571eb95e0de386e01470510c47def7444524b227597f9"
Sep 09 20:34:19.788955 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.788933    2547 scope.go:117] "RemoveContainer" containerID="992411527c349839e5a5cdac29fa68ca4c07e74a9afaf0938f8b60207a738f65"
Sep 09 20:34:19.799980 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.799958    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:34:19.805942 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:19.805917    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-97f7b-predictor-6f5d8d9788-zbckb"]
Sep 09 20:34:20.948604 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:20.948566    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" path="/var/lib/kubelet/pods/ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4/volumes"
Sep 09 20:34:22.769518 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:22.769488    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:34:22.769917 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:22.769582    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:34:22.770051 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:22.770026    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:22.770360 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:22.770330    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:27.683843 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:27.683801    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:27.684230 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:27.683801    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:32.770396 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:32.770358    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:32.770782 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:32.770358    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:37.684036 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:37.683991    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:34:37.684036 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:37.684018    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:42.770392 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:42.770358    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:42.770858 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:42.770358    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:47.683192 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:47.683144    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:34:47.684169 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:47.684144    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:34:52.770903 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:52.770868    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:34:52.771321 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:52.770865    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:34:57.683950 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:34:57.683918    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:35:02.770539 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:35:02.770497    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:35:02.771153 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:35:02.771134    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:35:12.771164 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:35:12.771132    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:43:39.680756 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.680725    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:43:39.681314 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.680937    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" containerID="cri-o://f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b" gracePeriod=30
Sep 09 20:43:39.681314 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.681005    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kube-rbac-proxy" containerID="cri-o://354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe" gracePeriod=30
Sep 09 20:43:39.709972 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.709943    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:43:39.710238 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.710207    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" containerID="cri-o://21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba" gracePeriod=30
Sep 09 20:43:39.710395 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.710241    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kube-rbac-proxy" containerID="cri-o://34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a" gracePeriod=30
Sep 09 20:43:39.999644 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.999558    2547 generic.go:356] "Generic (PLEG): container finished" podID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerID="354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe" exitCode=2
Sep 09 20:43:39.999644 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:39.999631    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerDied","Data":"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"}
Sep 09 20:43:40.001349 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:40.001323    2547 generic.go:356] "Generic (PLEG): container finished" podID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerID="34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a" exitCode=2
Sep 09 20:43:40.001477 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:40.001387    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerDied","Data":"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"}
Sep 09 20:43:42.765124 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:42.765090    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:43:42.765637 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:42.765091    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 09 20:43:42.771131 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:42.771101    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:43:42.864697 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:42.864672    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:43:42.887152 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:42.887128    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:43:43.014251 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.014161    2547 generic.go:356] "Generic (PLEG): container finished" podID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerID="f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b" exitCode=0
Sep 09 20:43:43.014251 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.014205    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"
Sep 09 20:43:43.014251 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.014229    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerDied","Data":"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"}
Sep 09 20:43:43.014565 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.014319    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" event={"ID":"06caf308-6682-41dd-a365-1c0a17cc0e6e","Type":"ContainerDied","Data":"c0f96c6ec3d44277d0e934f0d0ab91564ece53bfd24ce9e9470c350b3112795a"}
Sep 09 20:43:43.014565 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.014357    2547 scope.go:117] "RemoveContainer" containerID="354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"
Sep 09 20:43:43.015906 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.015883    2547 generic.go:356] "Generic (PLEG): container finished" podID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerID="21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba" exitCode=0
Sep 09 20:43:43.016005 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.015948    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerDied","Data":"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"}
Sep 09 20:43:43.016005 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.015955    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"
Sep 09 20:43:43.016005 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.015995    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh" event={"ID":"7ae33901-ff0d-4f8a-b058-c56f41c2f828","Type":"ContainerDied","Data":"26093c63e942db6d09d4dbd6097c384b63b2a095b486c77ebca6af3e1e04fb8a"}
Sep 09 20:43:43.019466 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019447    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config\") pod \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") "
Sep 09 20:43:43.019582 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019476    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config\") pod \"06caf308-6682-41dd-a365-1c0a17cc0e6e\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") "
Sep 09 20:43:43.019582 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019512    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls\") pod \"06caf308-6682-41dd-a365-1c0a17cc0e6e\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") "
Sep 09 20:43:43.019582 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019539    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") pod \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") "
Sep 09 20:43:43.019758 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019590    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtrp5\" (UniqueName: \"kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5\") pod \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\" (UID: \"7ae33901-ff0d-4f8a-b058-c56f41c2f828\") "
Sep 09 20:43:43.019818 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019800    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06caf308-6682-41dd-a365-1c0a17cc0e6e" (UID: "06caf308-6682-41dd-a365-1c0a17cc0e6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:43.019877 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019810    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ae33901-ff0d-4f8a-b058-c56f41c2f828" (UID: "7ae33901-ff0d-4f8a-b058-c56f41c2f828"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:43.019877 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.019833    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99wsf\" (UniqueName: \"kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf\") pod \"06caf308-6682-41dd-a365-1c0a17cc0e6e\" (UID: \"06caf308-6682-41dd-a365-1c0a17cc0e6e\") "
Sep 09 20:43:43.020473 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.020099    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.020473 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.020122    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.022021 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.021994    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf" (OuterVolumeSpecName: "kube-api-access-99wsf") pod "06caf308-6682-41dd-a365-1c0a17cc0e6e" (UID: "06caf308-6682-41dd-a365-1c0a17cc0e6e"). InnerVolumeSpecName "kube-api-access-99wsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:43.022146 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.022130    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ae33901-ff0d-4f8a-b058-c56f41c2f828" (UID: "7ae33901-ff0d-4f8a-b058-c56f41c2f828"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:43.022146 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.022136    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06caf308-6682-41dd-a365-1c0a17cc0e6e" (UID: "06caf308-6682-41dd-a365-1c0a17cc0e6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:43.022235 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.022208    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5" (OuterVolumeSpecName: "kube-api-access-rtrp5") pod "7ae33901-ff0d-4f8a-b058-c56f41c2f828" (UID: "7ae33901-ff0d-4f8a-b058-c56f41c2f828"). InnerVolumeSpecName "kube-api-access-rtrp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:43.022970 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.022955    2547 scope.go:117] "RemoveContainer" containerID="f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"
Sep 09 20:43:43.030009 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.029994    2547 scope.go:117] "RemoveContainer" containerID="354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"
Sep 09 20:43:43.030277 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:43:43.030258    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe\": container with ID starting with 354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe not found: ID does not exist" containerID="354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"
Sep 09 20:43:43.030349 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.030301    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe"} err="failed to get container status \"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe\": rpc error: code = NotFound desc = could not find container \"354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe\": container with ID starting with 354e84fbda8c358dffec1f69355e0b8ca53303ba4fa7651bc61e8b7e75f64dfe not found: ID does not exist"
Sep 09 20:43:43.030349 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.030322    2547 scope.go:117] "RemoveContainer" containerID="f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"
Sep 09 20:43:43.030592 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:43:43.030575    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b\": container with ID starting with f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b not found: ID does not exist" containerID="f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"
Sep 09 20:43:43.030629 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.030601    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b"} err="failed to get container status \"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b\": rpc error: code = NotFound desc = could not find container \"f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b\": container with ID starting with f2a11e3baf7acff70c33ebedf01443257d84af25b09035459d227d0fa673032b not found: ID does not exist"
Sep 09 20:43:43.030629 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.030620    2547 scope.go:117] "RemoveContainer" containerID="34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"
Sep 09 20:43:43.037404 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.037366    2547 scope.go:117] "RemoveContainer" containerID="21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"
Sep 09 20:43:43.043946 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.043931    2547 scope.go:117] "RemoveContainer" containerID="34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"
Sep 09 20:43:43.044186 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:43:43.044170    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a\": container with ID starting with 34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a not found: ID does not exist" containerID="34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"
Sep 09 20:43:43.044231 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.044192    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a"} err="failed to get container status \"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a\": rpc error: code = NotFound desc = could not find container \"34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a\": container with ID starting with 34f954cd8854dc163d5d70ad115f981f898b738f3c06da99542889d676d8193a not found: ID does not exist"
Sep 09 20:43:43.044231 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.044207    2547 scope.go:117] "RemoveContainer" containerID="21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"
Sep 09 20:43:43.044463 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:43:43.044448    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba\": container with ID starting with 21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba not found: ID does not exist" containerID="21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"
Sep 09 20:43:43.044505 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.044467    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba"} err="failed to get container status \"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba\": rpc error: code = NotFound desc = could not find container \"21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba\": container with ID starting with 21e72294396626e75f9105e4151611cd54ebfe0fb25f2bbd4ff272aadb3b4eba not found: ID does not exist"
Sep 09 20:43:43.120593 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.120559    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrp5\" (UniqueName: \"kubernetes.io/projected/7ae33901-ff0d-4f8a-b058-c56f41c2f828-kube-api-access-rtrp5\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.120593 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.120585    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wsf\" (UniqueName: \"kubernetes.io/projected/06caf308-6682-41dd-a365-1c0a17cc0e6e-kube-api-access-99wsf\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.120593 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.120596    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06caf308-6682-41dd-a365-1c0a17cc0e6e-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.120821 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.120605    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ae33901-ff0d-4f8a-b058-c56f41c2f828-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:43:43.338652 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.338629    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:43:43.345106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.345084    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj"]
Sep 09 20:43:43.354916 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.354891    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:43:43.358193 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.358174    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b9ccb-predictor-d5c8df47c-x72vh"]
Sep 09 20:43:43.771116 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:43.771072    2547 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b9ccb-predictor-869d5b846d-8ssrj" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: i/o timeout"
Sep 09 20:43:44.945665 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:44.945635    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" path="/var/lib/kubelet/pods/06caf308-6682-41dd-a365-1c0a17cc0e6e/volumes"
Sep 09 20:43:44.946034 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:43:44.946023    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" path="/var/lib/kubelet/pods/7ae33901-ff0d-4f8a-b058-c56f41c2f828/volumes"
Sep 09 20:51:29.775033 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.774860    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:51:29.777571 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.775140    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container" containerID="cri-o://41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1" gracePeriod=30
Sep 09 20:51:29.777571 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.775167    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kube-rbac-proxy" containerID="cri-o://435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7" gracePeriod=30
Sep 09 20:51:29.807870 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.807830    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:51:29.808154 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.808105    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container" containerID="cri-o://7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3" gracePeriod=30
Sep 09 20:51:29.808220 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:29.808165    2547 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kube-rbac-proxy" containerID="cri-o://680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d" gracePeriod=30
Sep 09 20:51:30.699354 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:30.699320    2547 generic.go:356] "Generic (PLEG): container finished" podID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerID="680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d" exitCode=2
Sep 09 20:51:30.699534 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:30.699392    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerDied","Data":"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"}
Sep 09 20:51:30.700971 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:30.700949    2547 generic.go:356] "Generic (PLEG): container finished" podID="e0879e79-177e-48a8-8992-e60a36b9898d" containerID="435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7" exitCode=2
Sep 09 20:51:30.701127 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:30.701001    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerDied","Data":"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"}
Sep 09 20:51:32.655106 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.655082    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:51:32.679449 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.679427    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:51:32.710000 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.709917    2547 generic.go:356] "Generic (PLEG): container finished" podID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerID="7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3" exitCode=0
Sep 09 20:51:32.710151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.710011    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerDied","Data":"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"}
Sep 09 20:51:32.710151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.710018    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"
Sep 09 20:51:32.710151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.710070    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn" event={"ID":"7150978e-3939-47c3-bc39-626cd5d3fb92","Type":"ContainerDied","Data":"d84a834fb921f176617b21d8590534f2df1ab605470613b4f830bbc75b17a387"}
Sep 09 20:51:32.710151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.710103    2547 scope.go:117] "RemoveContainer" containerID="680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"
Sep 09 20:51:32.711743 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.711718    2547 generic.go:356] "Generic (PLEG): container finished" podID="e0879e79-177e-48a8-8992-e60a36b9898d" containerID="41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1" exitCode=0
Sep 09 20:51:32.711860 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.711756    2547 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"
Sep 09 20:51:32.711860 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.711798    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerDied","Data":"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"}
Sep 09 20:51:32.711860 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.711844    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk" event={"ID":"e0879e79-177e-48a8-8992-e60a36b9898d","Type":"ContainerDied","Data":"24bbf0e5f9755f438263641b8199927a6f3d0f09b3fb995abba4a5e77300b6ca"}
Sep 09 20:51:32.718007 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.717989    2547 scope.go:117] "RemoveContainer" containerID="7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"
Sep 09 20:51:32.724768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.724750    2547 scope.go:117] "RemoveContainer" containerID="680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"
Sep 09 20:51:32.725009 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:51:32.724991    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d\": container with ID starting with 680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d not found: ID does not exist" containerID="680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"
Sep 09 20:51:32.725069 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.725018    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d"} err="failed to get container status \"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d\": rpc error: code = NotFound desc = could not find container \"680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d\": container with ID starting with 680ff4ce973971b972f2e445b34b0bcd8ca3f6a2c1635404cc601762a72c114d not found: ID does not exist"
Sep 09 20:51:32.725069 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.725034    2547 scope.go:117] "RemoveContainer" containerID="7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"
Sep 09 20:51:32.725315 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:51:32.725280    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3\": container with ID starting with 7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3 not found: ID does not exist" containerID="7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"
Sep 09 20:51:32.725366 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.725323    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3"} err="failed to get container status \"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3\": rpc error: code = NotFound desc = could not find container \"7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3\": container with ID starting with 7a22a573b525bddacdc122788378fc8791501461bdab008e1c52942873b5cbc3 not found: ID does not exist"
Sep 09 20:51:32.725366 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.725338    2547 scope.go:117] "RemoveContainer" containerID="435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"
Sep 09 20:51:32.727413 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.727399    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config\") pod \"e0879e79-177e-48a8-8992-e60a36b9898d\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") "
Sep 09 20:51:32.727466 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.727436    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls\") pod \"e0879e79-177e-48a8-8992-e60a36b9898d\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") "
Sep 09 20:51:32.727511 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.727476    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5c6b\" (UniqueName: \"kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b\") pod \"e0879e79-177e-48a8-8992-e60a36b9898d\" (UID: \"e0879e79-177e-48a8-8992-e60a36b9898d\") "
Sep 09 20:51:32.727887 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.727856    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0879e79-177e-48a8-8992-e60a36b9898d" (UID: "e0879e79-177e-48a8-8992-e60a36b9898d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:32.729570 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.729548    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b" (OuterVolumeSpecName: "kube-api-access-s5c6b") pod "e0879e79-177e-48a8-8992-e60a36b9898d" (UID: "e0879e79-177e-48a8-8992-e60a36b9898d"). InnerVolumeSpecName "kube-api-access-s5c6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:32.729689 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.729643    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0879e79-177e-48a8-8992-e60a36b9898d" (UID: "e0879e79-177e-48a8-8992-e60a36b9898d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:32.733299 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.733269    2547 scope.go:117] "RemoveContainer" containerID="41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"
Sep 09 20:51:32.740178 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.740154    2547 scope.go:117] "RemoveContainer" containerID="435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"
Sep 09 20:51:32.740480 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:51:32.740459    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7\": container with ID starting with 435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7 not found: ID does not exist" containerID="435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"
Sep 09 20:51:32.740541 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.740487    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7"} err="failed to get container status \"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7\": rpc error: code = NotFound desc = could not find container \"435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7\": container with ID starting with 435b6d090a2dbbe33e519ed571ef3bfdfde45a11b9fa51e977d778ee44ee13c7 not found: ID does not exist"
Sep 09 20:51:32.740541 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.740505    2547 scope.go:117] "RemoveContainer" containerID="41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"
Sep 09 20:51:32.740774 ip-10-0-136-11 kubenswrapper[2547]: E0909 20:51:32.740756    2547 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1\": container with ID starting with 41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1 not found: ID does not exist" containerID="41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"
Sep 09 20:51:32.740815 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.740779    2547 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1"} err="failed to get container status \"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1\": rpc error: code = NotFound desc = could not find container \"41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1\": container with ID starting with 41bc9292d36e06c76d4573c8be0e1d8e5d6516cd5b5f47f362172424f0bf57e1 not found: ID does not exist"
Sep 09 20:51:32.828348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828316    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config\") pod \"7150978e-3939-47c3-bc39-626cd5d3fb92\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") "
Sep 09 20:51:32.828348 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828354    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgqh4\" (UniqueName: \"kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4\") pod \"7150978e-3939-47c3-bc39-626cd5d3fb92\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") "
Sep 09 20:51:32.828555 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828411    2547 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls\") pod \"7150978e-3939-47c3-bc39-626cd5d3fb92\" (UID: \"7150978e-3939-47c3-bc39-626cd5d3fb92\") "
Sep 09 20:51:32.828621 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828604    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5c6b\" (UniqueName: \"kubernetes.io/projected/e0879e79-177e-48a8-8992-e60a36b9898d-kube-api-access-s5c6b\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:32.828661 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828631    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0879e79-177e-48a8-8992-e60a36b9898d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:32.828661 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828645    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0879e79-177e-48a8-8992-e60a36b9898d-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:32.828736 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.828698    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7150978e-3939-47c3-bc39-626cd5d3fb92" (UID: "7150978e-3939-47c3-bc39-626cd5d3fb92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:51:32.830477 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.830462    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7150978e-3939-47c3-bc39-626cd5d3fb92" (UID: "7150978e-3939-47c3-bc39-626cd5d3fb92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:32.830557 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.830535    2547 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4" (OuterVolumeSpecName: "kube-api-access-qgqh4") pod "7150978e-3939-47c3-bc39-626cd5d3fb92" (UID: "7150978e-3939-47c3-bc39-626cd5d3fb92"). InnerVolumeSpecName "kube-api-access-qgqh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:32.929274 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.929247    2547 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:32.929274 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.929268    2547 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgqh4\" (UniqueName: \"kubernetes.io/projected/7150978e-3939-47c3-bc39-626cd5d3fb92-kube-api-access-qgqh4\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:32.929453 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:32.929277    2547 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7150978e-3939-47c3-bc39-626cd5d3fb92-proxy-tls\") on node \"ip-10-0-136-11.ec2.internal\" DevicePath \"\""
Sep 09 20:51:33.026766 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:33.026699    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:51:33.030801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:33.030780    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-54686-predictor-5478745645-rdwpn"]
Sep 09 20:51:33.041651 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:33.041621    2547 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:51:33.046162 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:33.046140    2547 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-54686-predictor-5b5bdc6d9b-7tchk"]
Sep 09 20:51:34.945310 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:34.945256    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" path="/var/lib/kubelet/pods/7150978e-3939-47c3-bc39-626cd5d3fb92/volumes"
Sep 09 20:51:34.945707 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:51:34.945692    2547 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" path="/var/lib/kubelet/pods/e0879e79-177e-48a8-8992-e60a36b9898d/volumes"
Sep 09 20:52:01.978063 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:01.978036    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sm8vh_99c7cf30-f214-4557-87ce-05025c7000eb/global-pull-secret-syncer/0.log"
Sep 09 20:52:02.076257 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:02.076225    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rsnjr_5a10dc16-1c01-4fe5-9fff-1003d62c6ab0/konnectivity-agent/0.log"
Sep 09 20:52:02.170768 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:02.170738    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-11.ec2.internal_0a90e5cd9b3bb35e0b18f98e5a2e76f5/haproxy/0.log"
Sep 09 20:52:06.937801 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:06.937765    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvxmg_6163133f-7fcf-4903-89b2-b0c04b415f0d/node-exporter/0.log"
Sep 09 20:52:06.967815 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:06.967787    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvxmg_6163133f-7fcf-4903-89b2-b0c04b415f0d/kube-rbac-proxy/0.log"
Sep 09 20:52:06.994435 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:06.994411    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lvxmg_6163133f-7fcf-4903-89b2-b0c04b415f0d/init-textfile/0.log"
Sep 09 20:52:09.946965 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.946928    2547 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"]
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947185    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947196    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947210    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947215    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947223    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947228    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947236    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947241    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947247    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947252    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947260    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947265    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947277    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947282    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947307    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947313    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947320    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947325    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947333    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947338    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947345    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947350    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947356    2547 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kube-rbac-proxy"
Sep 09 20:52:09.947378 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947360    2547 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947401    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kserve-container"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947410    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947415    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947421    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947426    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a2f6b7-48cb-4a1e-a91c-3fc43aa6aa16" containerName="kserve-container"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947431    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947436    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947441    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddeb1746-bbcc-463f-a3f3-dd7d089e0cb4" containerName="kserve-container"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947447    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="7150978e-3939-47c3-bc39-626cd5d3fb92" containerName="kube-rbac-proxy"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947451    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="06caf308-6682-41dd-a365-1c0a17cc0e6e" containerName="kserve-container"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947456    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae33901-ff0d-4f8a-b058-c56f41c2f828" containerName="kserve-container"
Sep 09 20:52:09.948097 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.947461    2547 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0879e79-177e-48a8-8992-e60a36b9898d" containerName="kserve-container"
Sep 09 20:52:09.951636 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.951619    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:09.954150 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.954124    2547 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qzq9p\"/\"default-dockercfg-7lvqh\""
Sep 09 20:52:09.954305 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.954269    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzq9p\"/\"openshift-service-ca.crt\""
Sep 09 20:52:09.954635 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.954622    2547 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzq9p\"/\"kube-root-ca.crt\""
Sep 09 20:52:09.962762 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.962741    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"]
Sep 09 20:52:09.995719 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.995691    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-proc\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:09.995851 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.995723    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-podres\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:09.995851 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.995756    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj88x\" (UniqueName: \"kubernetes.io/projected/4f7e1079-530f-4955-88ae-c64d564c06c6-kube-api-access-nj88x\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:09.995851 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.995822    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-lib-modules\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:09.995960 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:09.995882    2547 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-sys\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096643 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096609    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-sys\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096661    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-proc\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096686    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-podres\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096714    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj88x\" (UniqueName: \"kubernetes.io/projected/4f7e1079-530f-4955-88ae-c64d564c06c6-kube-api-access-nj88x\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096739    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-sys\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096760    2547 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-lib-modules\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.096799 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096741    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-proc\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.097041 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096850    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-podres\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.097041 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.096862    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f7e1079-530f-4955-88ae-c64d564c06c6-lib-modules\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.105933 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.105906    2547 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj88x\" (UniqueName: \"kubernetes.io/projected/4f7e1079-530f-4955-88ae-c64d564c06c6-kube-api-access-nj88x\") pod \"perf-node-gather-daemonset-qcqsm\" (UID: \"4f7e1079-530f-4955-88ae-c64d564c06c6\") " pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.261151 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.261054    2547 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.534306 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.534154    2547 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"]
Sep 09 20:52:10.537203 ip-10-0-136-11 kubenswrapper[2547]: W0909 20:52:10.537171    2547 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4f7e1079_530f_4955_88ae_c64d564c06c6.slice/crio-9794edce0e2876e77ce77041566d4631bf09908a1f92f00f6f9f1b63ef6b3e97 WatchSource:0}: Error finding container 9794edce0e2876e77ce77041566d4631bf09908a1f92f00f6f9f1b63ef6b3e97: Status 404 returned error can't find the container with id 9794edce0e2876e77ce77041566d4631bf09908a1f92f00f6f9f1b63ef6b3e97
Sep 09 20:52:10.851073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.850984    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm" event={"ID":"4f7e1079-530f-4955-88ae-c64d564c06c6","Type":"ContainerStarted","Data":"1a5d1d16840103bf9f6479aa4c5dc251657fb9992aff889ab07b88196ec3637a"}
Sep 09 20:52:10.851073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.851048    2547 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm" event={"ID":"4f7e1079-530f-4955-88ae-c64d564c06c6","Type":"ContainerStarted","Data":"9794edce0e2876e77ce77041566d4631bf09908a1f92f00f6f9f1b63ef6b3e97"}
Sep 09 20:52:10.851277 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.851077    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:10.867808 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:10.867759    2547 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm" podStartSLOduration=1.867738198 podStartE2EDuration="1.867738198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:10.866963306 +0000 UTC m=+3430.437896044" watchObservedRunningTime="2026-09-09 20:52:10.867738198 +0000 UTC m=+3430.438670914"
Sep 09 20:52:11.479616 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:11.479560    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xhjbb_01869d12-9e3b-4439-84bb-173a4f515694/dns/0.log"
Sep 09 20:52:11.500528 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:11.500503    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xhjbb_01869d12-9e3b-4439-84bb-173a4f515694/kube-rbac-proxy/0.log"
Sep 09 20:52:11.523599 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:11.523581    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2297_0c874116-2f48-4afd-92dd-caf462ac9684/dns-node-resolver/0.log"
Sep 09 20:52:12.034455 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:12.034424    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-666d74f78c-8lxrt_475a33e1-d95e-4062-8b03-08aadda4bfb0/registry/0.log"
Sep 09 20:52:12.099277 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:12.099248    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-plptx_163aa480-c64c-49e6-aa38-b51aabaf7eb8/node-ca/0.log"
Sep 09 20:52:13.218260 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:13.218231    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zwrn5_0b82ff8a-392c-4e58-bbc4-f0ec10f08a2c/serve-healthcheck-canary/0.log"
Sep 09 20:52:13.697756 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:13.697729    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bclv_4a384fbb-4972-422d-bcc9-6b7033639596/kube-rbac-proxy/0.log"
Sep 09 20:52:13.717891 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:13.717872    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bclv_4a384fbb-4972-422d-bcc9-6b7033639596/exporter/0.log"
Sep 09 20:52:13.738759 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:13.738738    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bclv_4a384fbb-4972-422d-bcc9-6b7033639596/extractor/0.log"
Sep 09 20:52:16.108923 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:16.108893    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jpkkk_3d5011f8-6539-4059-ae7b-e49ee3bafaea/server/0.log"
Sep 09 20:52:16.192201 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:16.192173    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-z894z_f6688563-e9ec-4edb-99c2-8d115a3e67d2/manager/0.log"
Sep 09 20:52:16.277073 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:16.277053    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-7xgws_c180b994-b76b-4c36-b001-8208ee3e999a/seaweedfs/0.log"
Sep 09 20:52:16.864558 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:16.864533    2547 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qzq9p/perf-node-gather-daemonset-qcqsm"
Sep 09 20:52:22.205654 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.205626    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:52:22.227344 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.227322    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/egress-router-binary-copy/0.log"
Sep 09 20:52:22.247804 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.247762    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/cni-plugins/0.log"
Sep 09 20:52:22.267586 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.267566    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/bond-cni-plugin/0.log"
Sep 09 20:52:22.287749 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.287732    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/routeoverride-cni/0.log"
Sep 09 20:52:22.309033 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.309013    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/whereabouts-cni-bincopy/0.log"
Sep 09 20:52:22.330041 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.330015    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-svpxv_a70313ac-da9c-441e-89bb-da79eb54fdb8/whereabouts-cni/0.log"
Sep 09 20:52:22.425026 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.424979    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rx99p_32bab1db-f8ce-4f25-8b38-c48cfae58857/kube-multus/0.log"
Sep 09 20:52:22.534077 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.534002    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q9n5s_90d331aa-5052-4a95-91cc-b82896d0d9df/network-metrics-daemon/0.log"
Sep 09 20:52:22.553509 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:22.553485    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q9n5s_90d331aa-5052-4a95-91cc-b82896d0d9df/kube-rbac-proxy/0.log"
Sep 09 20:52:23.289174 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.289146    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/ovn-controller/0.log"
Sep 09 20:52:23.310096 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.310075    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/ovn-acl-logging/0.log"
Sep 09 20:52:23.339492 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.339471    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/kube-rbac-proxy-node/0.log"
Sep 09 20:52:23.363898 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.363875    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:52:23.380180 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.380157    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/northd/0.log"
Sep 09 20:52:23.399709 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.399692    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/nbdb/0.log"
Sep 09 20:52:23.420665 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.420642    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/sbdb/0.log"
Sep 09 20:52:23.467373 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:23.467355    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-26bsm_29ee8571-51bc-4fee-a100-6f72cb11057a/ovnkube-controller/0.log"
Sep 09 20:52:24.863039 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:24.863007    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cfnf2_f993fcdf-c198-419d-a123-b1daf4c877e9/network-check-target-container/0.log"
Sep 09 20:52:25.889490 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:25.889465    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7zmp2_16188a2b-afac-4c06-a671-f786c0937a3e/iptables-alerter/0.log"
Sep 09 20:52:26.587501 ip-10-0-136-11 kubenswrapper[2547]: I0909 20:52:26.587469    2547 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8qhwg_77b50dad-2bbc-42ab-8948-b921b410728a/tuned/0.log"