Jul 31 14:48:07.927537 ip-10-0-135-216 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:48:08.327319 ip-10-0-135-216 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:08.327319 ip-10-0-135-216 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:48:08.327319 ip-10-0-135-216 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:08.327319 ip-10-0-135-216 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:48:08.327319 ip-10-0-135-216 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:08.329650 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.329495    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337227    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337245    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337251    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337254    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337257    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.337251 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337260    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337265    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337269    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337272    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337275    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337277    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337280    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337283    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337286    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337289    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337291    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337294    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337297    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337300    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337302    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337305    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337307    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337310    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337312    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.337462 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337315    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337317    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337319    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337322    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337332    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337334    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337337    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337339    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337341    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337344    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337346    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337349    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337351    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337354    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337357    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337362    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337365    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337367    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337370    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337373    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.337912 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337375    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337378    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337380    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337383    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337385    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337388    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337390    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337393    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337395    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337398    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337400    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337403    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337405    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337408    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337411    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337413    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337416    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337418    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337421    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.338419 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337423    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337427    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337432    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337435    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337438    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337440    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337443    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337445    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337450    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337453    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337455    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337458    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337461    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337463    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337466    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337469    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337472    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337474    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337477    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337480    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.338877 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337484    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337486    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337489    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337492    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337495    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337499    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337502    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337506    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337509    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337511    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337513    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337516    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337518    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337520    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337523    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.337525    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338231    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338239    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338241    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.339380 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338245    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338249    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338252    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338255    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338258    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338262    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338266    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338269    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338272    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338275    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338277    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338280    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338283    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338285    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338288    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338290    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338293    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338296    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338298    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.339836 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338301    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338304    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338306    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338308    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338311    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338313    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338315    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338318    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338320    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338323    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338326    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338329    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338331    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338333    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338337    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338340    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338343    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338346    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338348    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338351    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.340349 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338354    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338356    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338359    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338363    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338367    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338369    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338373    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338377    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338381    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338384    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338387    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338389    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338392    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338394    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338397    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338399    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338401    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338404    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338407    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.340853 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338409    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338412    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338416    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338418    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338421    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338423    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338426    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338429    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338432    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338434    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338436    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338439    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338442    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338444    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338447    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338449    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338451    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338454    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338456    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338459    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.341339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338461    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338463    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338468    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338470    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338473    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338476    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338478    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338481    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338483    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338486    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338489    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338491    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338494    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338497    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338499    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338501    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338504    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338506    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338888    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338893    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.341829 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338896    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338899    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338902    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338905    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338907    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338910    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338913    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338915    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338918    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338920    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338923    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338925    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338928    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338930    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338933    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338936    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338938    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338941    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338943    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338946    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.342444 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338949    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338953    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338956    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338958    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338960    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338963    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338966    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338968    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338970    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338973    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338976    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338978    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338981    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338984    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338986    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338989    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338991    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338995    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.338999    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.342944 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339002    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339005    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339008    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339010    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339017    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339019    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339022    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339024    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339027    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339031    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339033    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339037    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339041    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339043    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339048    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339050    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339053    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339055    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.343436 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339058    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339061    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339063    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339065    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339068    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339071    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339074    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339077    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339080    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339083    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339086    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339088    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339090    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339093    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339095    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339098    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339100    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339102    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339105    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339107    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.343937 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339110    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339112    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339115    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339117    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339120    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339122    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339125    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339128    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339131    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339134    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339136    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339139    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339142    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339144    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339146    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339149    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339151    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339154    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339156    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339158    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339249    2568 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:48:08.344431 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339256    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339262    2568 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339266    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339271    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339274    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339278    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339282    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339285    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339288    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339291    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339295    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339298    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339301    2568 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339303    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339306    2568 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339310    2568 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339313    2568 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339317    2568 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339320    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339325    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339328    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339332    2568 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339335    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:48:08.344988 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339338    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339341    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339345    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339348    2568 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339350    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339353    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339356    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339359    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339363    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339366    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339369    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339371    2568 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339375    2568 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339377    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339382    2568 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339385    2568 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339388    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339391    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339394    2568 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339398    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339401    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339404    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339406    2568 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339409    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339412    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:48:08.345571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339415    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339418    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339421    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339425    2568 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339428    2568 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339432    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339434    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339437    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339440    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339443    2568 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339446    2568 flags.go:64] FLAG: --help="false"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339448    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339452    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339454    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339457    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339460    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339463    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339466    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339469    2568 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339472    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339474    2568 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339477    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339480    2568 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339482    2568 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:48:08.346233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339485    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339488    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339491    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339493    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339496    2568 flags.go:64] FLAG: --lock-file=""
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339498    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339501    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339504    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339509    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339512    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339515    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339520    2568 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339523    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339527    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339529    2568 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339533    2568 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339537    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339540    2568 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339544    2568 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339547    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339550    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339553    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339556    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339559    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339562    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:48:08.346803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339564    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339571    2568 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339574    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339577    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339580    2568 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339583    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339588    2568 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339591    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339594    2568 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339597    2568 flags.go:64] FLAG: --port="10250"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339600    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339603    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ad8ad5547dbea436"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339606    2568 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339609    2568 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339612    2568 flags.go:64] FLAG: --register-node="true"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339614    2568 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339618    2568 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339622    2568 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339625    2568 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339630    2568 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339634    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339637    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339640    2568 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339642    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339645    2568 flags.go:64] FLAG: --runonce="false"
Jul 31 14:48:08.347441 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339648    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339651    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339653    2568 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339656    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339659    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339662    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339665    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339668    2568 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339670    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339673    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339676    2568 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339679    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339681    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339684    2568 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339687    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339692    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339695    2568 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339697    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339701    2568 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339704    2568 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339706    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339709    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339711    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339714    2568 flags.go:64] FLAG: --v="2"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339718    2568 flags.go:64] FLAG: --version="false"
Jul 31 14:48:08.348041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339722    2568 flags.go:64] FLAG: --vmodule=""
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339728    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.339731    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339851    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339854    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339857    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339860    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339862    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339865    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339868    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339870    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339873    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339877    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339880    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339883    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339886    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339888    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339892    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339896    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.348801 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339899    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339902    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339904    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339906    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339909    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339911    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339914    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339916    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339919    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339921    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339924    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339926    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339929    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339931    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339937    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339939    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339942    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339944    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339947    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339949    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.349294 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339952    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339954    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339956    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339959    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339961    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339963    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339966    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339968    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339971    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339973    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339976    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339978    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339980    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339983    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339985    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339987    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339990    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339993    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339995    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.339997    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.349778 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340000    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340002    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340005    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340007    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340010    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340012    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340017    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340020    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340022    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340025    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340027    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340030    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340032    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340035    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340037    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340040    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340042    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340044    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340047    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340049    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.350271 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340052    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340054    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340057    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340059    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340062    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340064    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340067    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340069    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340071    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340074    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340076    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340079    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340081    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340083    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340086    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340088    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340092    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340094    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340100    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340104    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.350808 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340108    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340111    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.340114    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.340812    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.347452    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.347468    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.347487    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.347493    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347555    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347560    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347563    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347567    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347569    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347571    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347574    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.351307 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347576    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347579    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347581    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347584    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347586    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347589    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347592    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347595    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347597    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347600    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347602    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347605    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347609    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347613    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347616    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347620    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347623    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347627    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347629    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.351682 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347632    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347635    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347638    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347641    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347645    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347648    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347651    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347654    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347657    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347659    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347662    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347664    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347667    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347669    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347672    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347674    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347677    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347679    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347682    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.352357 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347684    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347687    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347689    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347692    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347694    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347696    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347699    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347701    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347704    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347707    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347710    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347712    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347715    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347718    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347720    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347722    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347725    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347728    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347731    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347733    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.352848 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347735    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347738    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347740    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347742    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347745    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347747    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347750    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347752    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347755    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347757    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347760    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347762    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347764    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347767    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347769    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347771    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347774    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347776    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347779    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347781    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.353394 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347783    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347786    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347788    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347791    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347793    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347796    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347798    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347801    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347804    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347807    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347810    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347812    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347816    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347820    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.347825    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:08.353910 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347947    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347952    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347955    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347958    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347961    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347963    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347966    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347969    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347972    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347974    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347976    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347979    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347981    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347983    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347986    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347988    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347991    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347993    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347996    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:08.354430 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.347998    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348001    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348003    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348006    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348052    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348056    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348058    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348061    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348064    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348067    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348069    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348072    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348074    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348077    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348079    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348082    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348085    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348089    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348092    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:08.355186 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348095    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348097    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348100    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348102    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348105    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348108    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348111    2568 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348114    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348116    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348118    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348121    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348124    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348126    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348129    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348131    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348134    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348136    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348138    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348141    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348144    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:08.355770 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348146    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348149    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348151    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348154    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348156    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348159    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348161    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348165    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348168    2568 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348171    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348173    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348176    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348179    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348181    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348184    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348187    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348190    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348210    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348215    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:08.356309 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348218    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348221    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348224    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348226    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348229    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348232    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348234    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348237    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348239    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348243    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348245    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348248    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348251    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348254    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348258    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348261    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348264    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348266    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348269    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:08.356804 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348271    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348274    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:08.348276    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.348281    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.348396    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.350288    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.351679    2568 server.go:1019] "Starting client certificate rotation"
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.351796    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.351848    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:08.357323 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.354114    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:08.359671 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.359650    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:08.368633 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.368614    2568 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:48:08.370552 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.370537    2568 log.go:25] "Validated CRI v1 image API"
Jul 31 14:48:08.371518 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.371499    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:48:08.376901 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.376875    2568 fs.go:135] Filesystem UUIDs: map[2ab23e72-59f4-49ac-99f0-ed95e42e4455:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c0a2b0bd-166e-4aaa-a8b7-6b429f12e132:/dev/nvme0n1p4]
Jul 31 14:48:08.376959 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.376902    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:48:08.383112 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.383003    2568 manager.go:216] Machine: {Timestamp:2026-07-31 14:48:08.381760261 +0000 UTC m=+0.336074161 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101767 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2796691574cbf8a8f3b1edafc207b6 SystemUUID:ec279669-1574-cbf8-a8f3-b1edafc207b6 BootID:b3972737-85c9-4037-a0cc-9b46621ad090 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a0:6b:0b:69:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a0:6b:0b:69:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:4d:fe:41:36:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:48:08.383112 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.383108    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:48:08.383245 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.383187    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:48:08.386507 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.386478    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:48:08.386642 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.386509    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-216.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:48:08.387264 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.387253    2568 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:48:08.387296 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.387266    2568 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:48:08.387296 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.387274    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:08.388054 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.388044    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:08.388746 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.388736    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:48:08.391851 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.391838    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:08.391960 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.391950    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:48:08.391992 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.391980    2568 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:48:08.392018 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.391994    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:48:08.392018 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392010    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:48:08.392018 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392019    2568 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:48:08.392098 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392028    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:48:08.392416 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392405    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:08.392460 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392423    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:08.392738 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.392722    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:08.394436 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.394421    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:48:08.397704 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.397682    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:48:08.397796 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.397729    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:48:08.399267 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399253    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399272    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399281    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399291    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399310    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399316    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399324    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399329    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399336    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399342    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399351    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:48:08.399371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399356    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:48:08.399984 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399976    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:48:08.399984 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.399984    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:48:08.403092 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.403078    2568 server.go:1268] "Started kubelet"
Jul 31 14:48:08.403234 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.403175    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:48:08.403368 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.403173    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:48:08.403428 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.403396    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:48:08.403962 ip-10-0-135-216 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:48:08.404405 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.404223    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-216.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:48:08.404405 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.404313    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:48:08.404405 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.404328    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:48:08.404710 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.404599    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:48:08.405568 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.405555    2568 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:48:08.408589 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.408568    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:48:08.409309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409289    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:48:08.409309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409296    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:08.409447 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.409368    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.409809 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409792    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:48:08.409809 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409832    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:48:08.409980 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409848    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:48:08.410052 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409986    2568 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:48:08.410052 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.409994    2568 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:48:08.411187 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.410188    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-216.ec2.internal.18c7673cde6cbc3a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-216.ec2.internal,UID:ip-10-0-135-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-216.ec2.internal,},FirstTimestamp:2026-07-31 14:48:08.403057722 +0000 UTC m=+0.357371618,LastTimestamp:2026-07-31 14:48:08.403057722 +0000 UTC m=+0.357371618,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-216.ec2.internal,}"
Jul 31 14:48:08.413966 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.413947    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:48:08.413966 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.413967    2568 factory.go:55] Registering systemd factory
Jul 31 14:48:08.414078 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.413975    2568 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:48:08.414342 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.414325    2568 factory.go:153] Registering CRI-O factory
Jul 31 14:48:08.414342 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.414341    2568 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:48:08.414489 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.414360    2568 factory.go:103] Registering Raw factory
Jul 31 14:48:08.414489 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.414369    2568 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:48:08.414731 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.414717    2568 manager.go:349] Starting recovery of all containers
Jul 31 14:48:08.420376 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.420343    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:48:08.420484 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.420446    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:48:08.426246 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.426228    2568 manager.go:354] Recovery completed
Jul 31 14:48:08.428866 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.428843    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z66sd"
Jul 31 14:48:08.430145 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.430130    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.432230 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432209    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.432311 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432252    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.432311 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432269    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.432849 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432833    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:48:08.432849 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432847    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:48:08.432966 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432866    2568 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:08.432966 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.432840    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.433655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.433641    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.433699 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.433672    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.433699 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.433682    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.434063 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.434051    2568 policy_none.go:49] "None policy: Start"
Jul 31 14:48:08.434102 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.434068    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:48:08.434102 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.434078    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:48:08.434369 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.434306    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-216.ec2.internal.18c7673ce029e2c1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-216.ec2.internal,UID:ip-10-0-135-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-216.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-216.ec2.internal,},FirstTimestamp:2026-07-31 14:48:08.432231105 +0000 UTC m=+0.386545008,LastTimestamp:2026-07-31 14:48:08.432231105 +0000 UTC m=+0.386545008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-216.ec2.internal,}"
Jul 31 14:48:08.435022 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.435011    2568 policy_none.go:47] "Start"
Jul 31 14:48:08.437949 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.437931    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z66sd"
Jul 31 14:48:08.484964 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.484802    2568 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:48:08.485059 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.485001    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:48:08.485059 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485019    2568 server.go:87] "Starting device plugin registration server"
Jul 31 14:48:08.485282 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485270    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:48:08.485330 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485288    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:48:08.485420 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485398    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:48:08.485512 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485500    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:48:08.485584 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.485513    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:48:08.486017 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.486000    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:48:08.486098 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.486052    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.582272 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.582171    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:48:08.583582 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.583566    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:48:08.583663 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.583612    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:48:08.583663 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.583641    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:48:08.583758 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.583683    2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:48:08.585371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.585353    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.586108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.586093    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:08.586561 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.586544    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.586655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.586580    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.586655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.586592    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.586655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.586613    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.598624 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.598606    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.598730 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.598631    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-216.ec2.internal\": node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.621488 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.621467    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.684446 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.684412    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"]
Jul 31 14:48:08.684561 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.684496    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.685503 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.685488    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.685571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.685519    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.685571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.685534    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.686773 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.686760    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.686895 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.686882    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.686947 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.686911    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.690780 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690766    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.690780 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690774    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.690896 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690795    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.690896 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690799    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.690896 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690809    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.691027 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.690810    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.692039 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.692026    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.692104 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.692050    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:08.692720 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.692705    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:08.692781 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.692738    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:08.692781 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.692774    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:08.711426 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.711381    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.711516 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.711429    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd551728a37b2cd396a15397d027448d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.ec2.internal\" (UID: \"bd551728a37b2cd396a15397d027448d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.711516 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.711445    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.719891 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.719873    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-216.ec2.internal\" not found" node="ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.721758 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.721741    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.724189 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.724175    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-216.ec2.internal\" not found" node="ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.811904 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811869    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.811904 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811905    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.812116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811929    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd551728a37b2cd396a15397d027448d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.ec2.internal\" (UID: \"bd551728a37b2cd396a15397d027448d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.812116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811960    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd551728a37b2cd396a15397d027448d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-216.ec2.internal\" (UID: \"bd551728a37b2cd396a15397d027448d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.812116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.812116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:08.811971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6621cd6dcb115baacbc5c444655fb55-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal\" (UID: \"f6621cd6dcb115baacbc5c444655fb55\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:08.822044 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.822024    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:08.922853 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:08.922792    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.022378 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.022326    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:09.023460 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.023442    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.027655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.027638    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:09.124241 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.124179    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.225058 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.224961    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.326042 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.326001    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.352153 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.352115    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:48:09.352670 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.352350    2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:48:09.426527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.426497    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.441382 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.441344    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:43:09 +0000 UTC" deadline="2028-03-18 11:55:32.44642496 +0000 UTC"
Jul 31 14:48:09.441442 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.441382    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14301h7m23.005047488s"
Jul 31 14:48:09.525135 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.525045    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:09.527240 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.527223    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.628171 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.628134    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.724839 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.724808    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:09.728262 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.728230    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.816917 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:09.816879    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd551728a37b2cd396a15397d027448d.slice/crio-2925eb96bfd032fb40fa06d067585a623f0d324c2ae0bc9cffa7d1283edf6095 WatchSource:0}: Error finding container 2925eb96bfd032fb40fa06d067585a623f0d324c2ae0bc9cffa7d1283edf6095: Status 404 returned error can't find the container with id 2925eb96bfd032fb40fa06d067585a623f0d324c2ae0bc9cffa7d1283edf6095
Jul 31 14:48:09.822601 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.822585    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:48:09.828410 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:09.828390    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-216.ec2.internal\" not found"
Jul 31 14:48:09.881431 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.881402    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:09.910344 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.910316    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:09.922525 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.922497    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:09.923714 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.923700    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal"
Jul 31 14:48:09.933384 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:09.933364    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:10.293034 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:10.292995    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6621cd6dcb115baacbc5c444655fb55.slice/crio-72bb4744e5fdf91676a19cdaa813ecc2dc2d88e3e69b47118cd04b60f4e0e580 WatchSource:0}: Error finding container 72bb4744e5fdf91676a19cdaa813ecc2dc2d88e3e69b47118cd04b60f4e0e580: Status 404 returned error can't find the container with id 72bb4744e5fdf91676a19cdaa813ecc2dc2d88e3e69b47118cd04b60f4e0e580
Jul 31 14:48:10.395559 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.395526    2568 apiserver.go:52] "Watching apiserver"
Jul 31 14:48:10.406693 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.406664    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:48:10.407768 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.407741    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-llt4j","openshift-dns/node-resolver-mh4z7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal","openshift-multus/multus-zp5fn","openshift-multus/network-metrics-daemon-lkpvs","openshift-network-diagnostics/network-check-target-zqnxm","kube-system/konnectivity-agent-wzpsj","kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal","openshift-image-registry/node-ca-rc6lj","openshift-multus/multus-additional-cni-plugins-ltrl7","openshift-network-operator/iptables-alerter-4hmd2","openshift-ovn-kubernetes/ovnkube-node-b95kt","kube-system/global-pull-secret-syncer-kbpzc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"]
Jul 31 14:48:10.410184 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.410154    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:10.411119 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.411097    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.413766 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.413743    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.413869 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.413828    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:10.414147 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.414122    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:48:10.414366 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.414351    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6pgft\""
Jul 31 14:48:10.414732 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.414554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:48:10.416037 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.416016    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.418487 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.418467    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.419487 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419467    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:48:10.419592 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419495    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.419748 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419727    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.419831 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419768    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6tqvx\""
Jul 31 14:48:10.419977 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419952    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4e8c6-8adc-4875-b2cf-4e4209776626-konnectivity-ca\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.420068 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.419995    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d4e8c6-8adc-4875-b2cf-4e4209776626-agent-certs\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.420768 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.420742    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.420874 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.420814    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:10.420963 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.420942    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.421074 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.421057    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-84vkd\""
Jul 31 14:48:10.421217 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.421202    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:48:10.421326 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.421223    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:48:10.421652 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.421630    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.422343 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.422284    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:10.423222 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.423201    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:10.423322 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.423291    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:10.426385 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.426365    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.429466 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.429441    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rh68j\""
Jul 31 14:48:10.429560 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.429500    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:48:10.429684 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.429665    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.429754 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.429682    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.429754 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.429705    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.432782 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.432757    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6qvcc\""
Jul 31 14:48:10.432782 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.432751    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.433086 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.433051    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.433679 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.433651    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.435482 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.435464    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x7fl2\""
Jul 31 14:48:10.435696 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.435673    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.435696 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.435690    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.436339 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.436316    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.438640 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.438621    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-svgjl\""
Jul 31 14:48:10.438869 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.438846    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.439714 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.439691    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:48:10.440224 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.440203    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:48:10.441953 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.441929    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.442499 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.442479    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:48:10.442793 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.442772    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fj6w9\""
Jul 31 14:48:10.443386 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.443115    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.445117 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.445100    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.447733 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.447686    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:48:10.448084 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.448061    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:48:10.449452 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.448868    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:48:10.449452 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.449060    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:48:10.449452 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.449252    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckbcj\""
Jul 31 14:48:10.449629 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.449499    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:48:10.449708 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.449691    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:48:10.450006 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.449970    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcxq2"
Jul 31 14:48:10.460251 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.458583    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcxq2"
Jul 31 14:48:10.511816 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.511792    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520345    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwsr\" (UniqueName: \"kubernetes.io/projected/9680aba1-e848-493d-929a-02fa6fc70f42-kube-api-access-fpwsr\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4e8c6-8adc-4875-b2cf-4e4209776626-konnectivity-ca\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-netns\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520434    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520452    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-systemd\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.520469 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-socket-dir-parent\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520495    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-daemon-config\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-bin\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520543    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovn-node-metrics-cert\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-system-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520581    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-k8s-cni-cncf-io\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520620    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-hostroot\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-systemd-units\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-env-overrides\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520687    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520711    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fr5\" (UniqueName: \"kubernetes.io/projected/f4556428-410f-4ec3-90ea-2257ae552b7a-kube-api-access-v4fr5\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520739    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-kubernetes\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.520792 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520765    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-conf\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520795    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-lib-modules\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520834    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e130e647-10ab-4777-a1ff-a3c083c60005-tmp-dir\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520861    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-iptables-alerter-script\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520886    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-os-release\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520902    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-kubelet\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520917    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-etc-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-ovn\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520962    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-kubelet-config\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520977    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d003d921-8080-41f5-a0fa-65086d2e7f14-serviceca\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.520993    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-device-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521017    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-conf-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521057    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblkp\" (UniqueName: \"kubernetes.io/projected/e130e647-10ab-4777-a1ff-a3c083c60005-kube-api-access-mblkp\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521096    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.521354 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d4e8c6-8adc-4875-b2cf-4e4209776626-agent-certs\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521136    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-registration-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521151    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-cni-binary-copy\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521167    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-netns\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521182    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d466d\" (UniqueName: \"kubernetes.io/projected/d003d921-8080-41f5-a0fa-65086d2e7f14-kube-api-access-d466d\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521198    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzxdm\" (UniqueName: \"kubernetes.io/projected/7bdf2867-f4e8-43a2-b790-61b9b5591436-kube-api-access-jzxdm\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cnibin\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521263    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521403    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-var-lib-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521452    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-netd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-config\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521522    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-etc-kubernetes\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521553    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521586    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521611    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-tmp\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.522108 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521628    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-host-slash\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-modprobe-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521723    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521754    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-host\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521776    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-tuned\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521792    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e130e647-10ab-4777-a1ff-a3c083c60005-hosts-file\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521865    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-log-socket\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521888    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-sys\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thcx2\" (UniqueName: \"kubernetes.io/projected/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-kube-api-access-thcx2\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521919    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3d4e8c6-8adc-4875-b2cf-4e4209776626-konnectivity-ca\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521938    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-socket-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521968    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9sg\" (UniqueName: \"kubernetes.io/projected/3ad1e27e-0327-47e2-8dad-8416e7d99494-kube-api-access-cl9sg\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.521993    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-kubelet\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522042    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522083    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-bin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.522922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-system-cni-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-systemd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522165    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-node-log\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522190    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d003d921-8080-41f5-a0fa-65086d2e7f14-host\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522233    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysconfig\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522257    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-run\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522282    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-sys-fs\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522307    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-multus\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522333    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8z9\" (UniqueName: \"kubernetes.io/projected/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-kube-api-access-bg8z9\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522359    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-script-lib\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522382    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-var-lib-kubelet\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522408    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-cnibin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522427    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-multus-certs\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-os-release\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-slash\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522497    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:10.523749 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522526    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbg95\" (UniqueName: \"kubernetes.io/projected/d52edadf-6b08-4eda-8a43-c5d889d953fe-kube-api-access-nbg95\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.524495 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.522549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-dbus\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.528581 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.528559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3d4e8c6-8adc-4875-b2cf-4e4209776626-agent-certs\") pod \"konnectivity-agent-wzpsj\" (UID: \"e3d4e8c6-8adc-4875-b2cf-4e4209776626\") " pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.538429 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.538399    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:10.591468 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.591320    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal" event={"ID":"f6621cd6dcb115baacbc5c444655fb55","Type":"ContainerStarted","Data":"72bb4744e5fdf91676a19cdaa813ecc2dc2d88e3e69b47118cd04b60f4e0e580"}
Jul 31 14:48:10.594346 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.594317    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal" event={"ID":"bd551728a37b2cd396a15397d027448d","Type":"ContainerStarted","Data":"2925eb96bfd032fb40fa06d067585a623f0d324c2ae0bc9cffa7d1283edf6095"}
Jul 31 14:48:10.623827 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-conf-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623833    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623861    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblkp\" (UniqueName: \"kubernetes.io/projected/e130e647-10ab-4777-a1ff-a3c083c60005-kube-api-access-mblkp\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623891    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623916    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-registration-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623921    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-conf-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623937    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-cni-binary-copy\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.623976    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-netns\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.624014 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d466d\" (UniqueName: \"kubernetes.io/projected/d003d921-8080-41f5-a0fa-65086d2e7f14-kube-api-access-d466d\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624024    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzxdm\" (UniqueName: \"kubernetes.io/projected/7bdf2867-f4e8-43a2-b790-61b9b5591436-kube-api-access-jzxdm\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624058    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cnibin\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cnibin\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624146    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-netns\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624465    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.624492 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.624494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-var-lib-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.625320 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.625290    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-netd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.625454 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.625352    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-config\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.625454 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.625382    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-etc-kubernetes\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.625454 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.625417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.625610 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.625452    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627151    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627069    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-registration-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627159    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-tmp\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627367    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-host-slash\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627355    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-var-lib-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.627525    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627532    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627553    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627661    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-host-slash\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627730    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-cni-binary-copy\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-modprobe-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627841    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-host\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.627963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-tuned\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628299    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-config\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628419    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-etc-kubernetes\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628486    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-netd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628555    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e130e647-10ab-4777-a1ff-a3c083c60005-hosts-file\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.630480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628605    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-log-socket\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.628965    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-host\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629014    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e130e647-10ab-4777-a1ff-a3c083c60005-hosts-file\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-log-socket\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629180    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-sys\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629388    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thcx2\" (UniqueName: \"kubernetes.io/projected/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-kube-api-access-thcx2\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.629410    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.129378264 +0000 UTC m=+3.083692176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629464    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-socket-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629658    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9sg\" (UniqueName: \"kubernetes.io/projected/3ad1e27e-0327-47e2-8dad-8416e7d99494-kube-api-access-cl9sg\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629690    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-kubelet\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.629922    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-bin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630096    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-sys\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630105    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-system-cni-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.631526 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630137    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630175    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-systemd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630197    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-node-log\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630204    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630251    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-socket-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-system-cni-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630444    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-kubelet\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630469    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-node-log\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630507    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-bin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630647    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d003d921-8080-41f5-a0fa-65086d2e7f14-host\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-systemd\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysconfig\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630745    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-run\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d003d921-8080-41f5-a0fa-65086d2e7f14-host\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630780    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-sys-fs\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630857    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-multus\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630907    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-run\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630764    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysconfig\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.632372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630943    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-sys-fs\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8z9\" (UniqueName: \"kubernetes.io/projected/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-kube-api-access-bg8z9\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630995    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-script-lib\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631020    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-var-lib-kubelet\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.630997    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-var-lib-cni-multus\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631051    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-cnibin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631116    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-cnibin\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631139    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-multus-certs\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631184    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-var-lib-kubelet\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631226    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-os-release\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631397    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-multus-certs\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631313    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-slash\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631515    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-os-release\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631556    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-modprobe-d\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631937    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-tuned\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632068    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-slash\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.633201 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.631520    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbg95\" (UniqueName: \"kubernetes.io/projected/d52edadf-6b08-4eda-8a43-c5d889d953fe-kube-api-access-nbg95\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632123    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-dbus\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632338    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwsr\" (UniqueName: \"kubernetes.io/projected/9680aba1-e848-493d-929a-02fa6fc70f42-kube-api-access-fpwsr\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632407    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-netns\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632445    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-systemd\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632661    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-dbus\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632731    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-socket-dir-parent\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.632992    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-netns\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633032    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-daemon-config\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633066    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-bin\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633098    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-socket-dir-parent\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633211    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-systemd\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633234    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovn-node-metrics-cert\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633414    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-system-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633495    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-k8s-cni-cncf-io\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633626    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-hostroot\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633709    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-systemd-units\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633739    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-env-overrides\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633777    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4556428-410f-4ec3-90ea-2257ae552b7a-tmp\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633769    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633857    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fr5\" (UniqueName: \"kubernetes.io/projected/f4556428-410f-4ec3-90ea-2257ae552b7a-kube-api-access-v4fr5\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633925    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-daemon-config\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633929    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-kubernetes\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.633998    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-conf\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634035    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-lib-modules\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634082    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.634066    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-host-run-k8s-cni-cncf-io\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.634736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634330    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-lib-modules\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.634847    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.134264551 +0000 UTC m=+3.088578441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634862    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-hostroot\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.634894    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-kubernetes\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635027    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-system-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-cni-bin\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635231 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635228    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-multus-cni-dir\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635256    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e130e647-10ab-4777-a1ff-a3c083c60005-tmp-dir\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-iptables-alerter-script\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635297    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4556428-410f-4ec3-90ea-2257ae552b7a-etc-sysctl-conf\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635331    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-os-release\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635395    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-env-overrides\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635511    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-kubelet\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635529    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7bdf2867-f4e8-43a2-b790-61b9b5591436-os-release\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635545    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-etc-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635585    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovnkube-script-lib\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635598    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-etc-openvswitch\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635601    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e130e647-10ab-4777-a1ff-a3c083c60005-tmp-dir\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635609    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-ovn\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635640    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-kubelet-config\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.635656 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635658    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-systemd-units\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635657    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-run-ovn\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635586    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d52edadf-6b08-4eda-8a43-c5d889d953fe-host-kubelet\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635693    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d003d921-8080-41f5-a0fa-65086d2e7f14-serviceca\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-device-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635859    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-etc-selinux\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635870    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0ab4de6-e675-43b2-9930-5c43ad601b61-kubelet-config\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.635940    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ad1e27e-0327-47e2-8dad-8416e7d99494-device-dir\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.636444 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.636160    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d003d921-8080-41f5-a0fa-65086d2e7f14-serviceca\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.636823 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.636525    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-iptables-alerter-script\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.636823 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.636601    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d466d\" (UniqueName: \"kubernetes.io/projected/d003d921-8080-41f5-a0fa-65086d2e7f14-kube-api-access-d466d\") pod \"node-ca-rc6lj\" (UID: \"d003d921-8080-41f5-a0fa-65086d2e7f14\") " pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.637706 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.637679    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d52edadf-6b08-4eda-8a43-c5d889d953fe-ovn-node-metrics-cert\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.638493 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.638458    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:10.638493 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.638481    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:10.638493 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.638495    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:10.638681 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.638589    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblkp\" (UniqueName: \"kubernetes.io/projected/e130e647-10ab-4777-a1ff-a3c083c60005-kube-api-access-mblkp\") pod \"node-resolver-mh4z7\" (UID: \"e130e647-10ab-4777-a1ff-a3c083c60005\") " pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.638681 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:10.638637    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:11.138610661 +0000 UTC m=+3.092924562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:10.638809 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.638787    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzxdm\" (UniqueName: \"kubernetes.io/projected/7bdf2867-f4e8-43a2-b790-61b9b5591436-kube-api-access-jzxdm\") pod \"multus-zp5fn\" (UID: \"7bdf2867-f4e8-43a2-b790-61b9b5591436\") " pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.639311 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.639249    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thcx2\" (UniqueName: \"kubernetes.io/projected/b810a9e0-78e8-41f9-9241-c49de4bdb5b5-kube-api-access-thcx2\") pod \"iptables-alerter-4hmd2\" (UID: \"b810a9e0-78e8-41f9-9241-c49de4bdb5b5\") " pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.639726 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.639705    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8z9\" (UniqueName: \"kubernetes.io/projected/0888bf70-75f8-4ecb-8d07-db4ddceb29c2-kube-api-access-bg8z9\") pod \"multus-additional-cni-plugins-ltrl7\" (UID: \"0888bf70-75f8-4ecb-8d07-db4ddceb29c2\") " pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.641453 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.641429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9sg\" (UniqueName: \"kubernetes.io/projected/3ad1e27e-0327-47e2-8dad-8416e7d99494-kube-api-access-cl9sg\") pod \"aws-ebs-csi-driver-node-6x2d4\" (UID: \"3ad1e27e-0327-47e2-8dad-8416e7d99494\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.646980 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.646954    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fr5\" (UniqueName: \"kubernetes.io/projected/f4556428-410f-4ec3-90ea-2257ae552b7a-kube-api-access-v4fr5\") pod \"tuned-llt4j\" (UID: \"f4556428-410f-4ec3-90ea-2257ae552b7a\") " pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.647562 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.647541    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwsr\" (UniqueName: \"kubernetes.io/projected/9680aba1-e848-493d-929a-02fa6fc70f42-kube-api-access-fpwsr\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:10.647660 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.647616    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbg95\" (UniqueName: \"kubernetes.io/projected/d52edadf-6b08-4eda-8a43-c5d889d953fe-kube-api-access-nbg95\") pod \"ovnkube-node-b95kt\" (UID: \"d52edadf-6b08-4eda-8a43-c5d889d953fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:10.725378 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.725340    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:10.733803 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.733564    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4"
Jul 31 14:48:10.736060 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:10.736027    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d4e8c6_8adc_4875_b2cf_4e4209776626.slice/crio-deed9992149bcfd5d7c7f96ed916caa3acdfd61738d62f314ff877c1057ece2f WatchSource:0}: Error finding container deed9992149bcfd5d7c7f96ed916caa3acdfd61738d62f314ff877c1057ece2f: Status 404 returned error can't find the container with id deed9992149bcfd5d7c7f96ed916caa3acdfd61738d62f314ff877c1057ece2f
Jul 31 14:48:10.743451 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.743423    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zp5fn"
Jul 31 14:48:10.745513 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:10.745487    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad1e27e_0327_47e2_8dad_8416e7d99494.slice/crio-4f0565a051e8d70a1746f10e5bf8ae0ce1469102f8fffe08246a7874e4b7283e WatchSource:0}: Error finding container 4f0565a051e8d70a1746f10e5bf8ae0ce1469102f8fffe08246a7874e4b7283e: Status 404 returned error can't find the container with id 4f0565a051e8d70a1746f10e5bf8ae0ce1469102f8fffe08246a7874e4b7283e
Jul 31 14:48:10.752455 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.752430    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rc6lj"
Jul 31 14:48:10.761297 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.761268    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-llt4j"
Jul 31 14:48:10.768468 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.768446    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mh4z7"
Jul 31 14:48:10.776056 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.776029    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ltrl7"
Jul 31 14:48:10.786207 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.786095    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hmd2"
Jul 31 14:48:10.793648 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:10.793622    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:11.138840 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.138802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:11.139038 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.138865    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:11.139038 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.138902    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:11.139161 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139034    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:11.139161 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139105    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.139082212 +0000 UTC m=+4.093396116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:11.139593 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139568    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:11.139727 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139629    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.139611096 +0000 UTC m=+4.093924985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:11.139802 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139738    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:11.139802 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139755    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:11.139802 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139768    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:11.139935 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.139809    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:12.139795742 +0000 UTC m=+4.094109643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:11.183157 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:11.183128    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdf2867_f4e8_43a2_b790_61b9b5591436.slice/crio-34f8d946a1cc8b985c8e3c466eb9ffe201020dd4bde2a53a20d5f44659083fb3 WatchSource:0}: Error finding container 34f8d946a1cc8b985c8e3c466eb9ffe201020dd4bde2a53a20d5f44659083fb3: Status 404 returned error can't find the container with id 34f8d946a1cc8b985c8e3c466eb9ffe201020dd4bde2a53a20d5f44659083fb3
Jul 31 14:48:11.462622 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.462526    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:11 +0000 UTC" deadline="2028-01-08 08:00:41.650642523 +0000 UTC"
Jul 31 14:48:11.462622 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.462562    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12617h12m30.188085121s"
Jul 31 14:48:11.584714 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.584682    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:11.584896 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:11.584837    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:11.598829 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.598704    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4" event={"ID":"3ad1e27e-0327-47e2-8dad-8416e7d99494","Type":"ContainerStarted","Data":"4f0565a051e8d70a1746f10e5bf8ae0ce1469102f8fffe08246a7874e4b7283e"}
Jul 31 14:48:11.600544 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.600506    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wzpsj" event={"ID":"e3d4e8c6-8adc-4875-b2cf-4e4209776626","Type":"ContainerStarted","Data":"deed9992149bcfd5d7c7f96ed916caa3acdfd61738d62f314ff877c1057ece2f"}
Jul 31 14:48:11.602153 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:11.602118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zp5fn" event={"ID":"7bdf2867-f4e8-43a2-b790-61b9b5591436","Type":"ContainerStarted","Data":"34f8d946a1cc8b985c8e3c466eb9ffe201020dd4bde2a53a20d5f44659083fb3"}
Jul 31 14:48:12.117128 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:12.115326    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0888bf70_75f8_4ecb_8d07_db4ddceb29c2.slice/crio-d1b976d13f86d5259f0d83a918b770b9fc19f8b7757ea49839dbaf187fd9d339 WatchSource:0}: Error finding container d1b976d13f86d5259f0d83a918b770b9fc19f8b7757ea49839dbaf187fd9d339: Status 404 returned error can't find the container with id d1b976d13f86d5259f0d83a918b770b9fc19f8b7757ea49839dbaf187fd9d339
Jul 31 14:48:12.122661 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:12.121733    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode130e647_10ab_4777_a1ff_a3c083c60005.slice/crio-bcb8e65e1c1fca4b7c78d159ddb9fe3f35a19763f11be845d112c3d36f167256 WatchSource:0}: Error finding container bcb8e65e1c1fca4b7c78d159ddb9fe3f35a19763f11be845d112c3d36f167256: Status 404 returned error can't find the container with id bcb8e65e1c1fca4b7c78d159ddb9fe3f35a19763f11be845d112c3d36f167256
Jul 31 14:48:12.124007 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:12.123812    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52edadf_6b08_4eda_8a43_c5d889d953fe.slice/crio-d21497f895917204e8a3b3db403ce4f3ede615dac284b1ab8c7d3f2cb59c0a5d WatchSource:0}: Error finding container d21497f895917204e8a3b3db403ce4f3ede615dac284b1ab8c7d3f2cb59c0a5d: Status 404 returned error can't find the container with id d21497f895917204e8a3b3db403ce4f3ede615dac284b1ab8c7d3f2cb59c0a5d
Jul 31 14:48:12.127433 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:48:12.126287    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd003d921_8080_41f5_a0fa_65086d2e7f14.slice/crio-8735b163c5076026dc9b01e6214207a45f07f1e7ba30791f2956a311c75190f8 WatchSource:0}: Error finding container 8735b163c5076026dc9b01e6214207a45f07f1e7ba30791f2956a311c75190f8: Status 404 returned error can't find the container with id 8735b163c5076026dc9b01e6214207a45f07f1e7ba30791f2956a311c75190f8
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.150625    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.150685    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.150720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150792    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150821    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150835    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150867    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150896    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.150875117 +0000 UTC m=+6.105189025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150916    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.15090619 +0000 UTC m=+6.105220082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150942    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:12.152527 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.150985    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.150968125 +0000 UTC m=+6.105282013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:12.463767 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.463420    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:11 +0000 UTC" deadline="2028-04-17 13:50:55.968189028 +0000 UTC"
Jul 31 14:48:12.463767 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.463658    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15023h2m43.504539064s"
Jul 31 14:48:12.584770 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.584737    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:12.584770 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.584761    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:12.585021 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.584884    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:12.585083 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:12.585029    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:12.605482 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.605445    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rc6lj" event={"ID":"d003d921-8080-41f5-a0fa-65086d2e7f14","Type":"ContainerStarted","Data":"8735b163c5076026dc9b01e6214207a45f07f1e7ba30791f2956a311c75190f8"}
Jul 31 14:48:12.607285 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.606853    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"d21497f895917204e8a3b3db403ce4f3ede615dac284b1ab8c7d3f2cb59c0a5d"}
Jul 31 14:48:12.608181 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.608156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mh4z7" event={"ID":"e130e647-10ab-4777-a1ff-a3c083c60005","Type":"ContainerStarted","Data":"bcb8e65e1c1fca4b7c78d159ddb9fe3f35a19763f11be845d112c3d36f167256"}
Jul 31 14:48:12.609822 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.609460    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-llt4j" event={"ID":"f4556428-410f-4ec3-90ea-2257ae552b7a","Type":"ContainerStarted","Data":"a0cdf7ce2810facf09a61efd21e4c588a760d655a72115dff82040727320a065"}
Jul 31 14:48:12.612387 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.612360    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal" event={"ID":"bd551728a37b2cd396a15397d027448d","Type":"ContainerStarted","Data":"078633d6233094b85abe4be53b47c8a91c4b7f8ba2f4f7ab1f6e832505ce2579"}
Jul 31 14:48:12.614081 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.614057    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hmd2" event={"ID":"b810a9e0-78e8-41f9-9241-c49de4bdb5b5","Type":"ContainerStarted","Data":"b954bcf50553cd6bc76295d6cab13112e6761c77a3aeb1e42fbd997b991cac0d"}
Jul 31 14:48:12.615489 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.615458    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerStarted","Data":"d1b976d13f86d5259f0d83a918b770b9fc19f8b7757ea49839dbaf187fd9d339"}
Jul 31 14:48:12.621628 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.621600    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal" event={"ID":"f6621cd6dcb115baacbc5c444655fb55","Type":"ContainerStarted","Data":"dbe806ece026382aabd55600da59b6f84bf975880332fa48718fb22f6495c0e7"}
Jul 31 14:48:12.661998 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:12.661882    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-216.ec2.internal" podStartSLOduration=2.661858864 podStartE2EDuration="2.661858864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:12.629662081 +0000 UTC m=+4.583975990" watchObservedRunningTime="2026-07-31 14:48:12.661858864 +0000 UTC m=+4.616172777"
Jul 31 14:48:13.584639 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:13.584570    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:13.585300 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:13.584754    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:13.626657 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:13.626594    2568 generic.go:356] "Generic (PLEG): container finished" podID="f6621cd6dcb115baacbc5c444655fb55" containerID="dbe806ece026382aabd55600da59b6f84bf975880332fa48718fb22f6495c0e7" exitCode=0
Jul 31 14:48:13.627736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:13.627705    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal" event={"ID":"f6621cd6dcb115baacbc5c444655fb55","Type":"ContainerDied","Data":"dbe806ece026382aabd55600da59b6f84bf975880332fa48718fb22f6495c0e7"}
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.170000    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.170057    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.170107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170244    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170325    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.170304668 +0000 UTC m=+10.124618563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170759    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170782    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170797    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170847    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.1708311 +0000 UTC m=+10.125145004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170927    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:14.171012 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.170965    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:18.170953273 +0000 UTC m=+10.125267159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:14.585524 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.585414    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:14.586059 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.585560    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:14.586059 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.585415    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:14.586059 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:14.586012    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:14.636090 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.635258    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wzpsj" event={"ID":"e3d4e8c6-8adc-4875-b2cf-4e4209776626","Type":"ContainerStarted","Data":"40e9965049ee5aabdec11fe69d13c165c4ef07c3441c7f20feaa614a9de3013f"}
Jul 31 14:48:14.654630 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.654217    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wzpsj" podStartSLOduration=2.373544512 podStartE2EDuration="5.654195951s" totalImagesPullingTime="3.280651439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:14.65321054 +0000 UTC m=+6.607524450" watchObservedRunningTime="2026-07-31 14:48:14.654195951 +0000 UTC m=+6.608509861"
Jul 31 14:48:14.658355 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.658288    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal" event={"ID":"f6621cd6dcb115baacbc5c444655fb55","Type":"ContainerStarted","Data":"520fbdb8866d8032e3ae6a376ff18084f3800a08d01d3b527e8c02ff0e28b266"}
Jul 31 14:48:14.666711 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.666667    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4" event={"ID":"3ad1e27e-0327-47e2-8dad-8416e7d99494","Type":"ContainerStarted","Data":"99e2311f10935f46f9bb4423dc79d7cceb3db872a3aeef04a047af795c659c04"}
Jul 31 14:48:14.678673 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:14.678616    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-216.ec2.internal" podStartSLOduration=4.678596727 podStartE2EDuration="4.678596727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:14.677969192 +0000 UTC m=+6.632283105" watchObservedRunningTime="2026-07-31 14:48:14.678596727 +0000 UTC m=+6.632910639"
Jul 31 14:48:15.578333 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:15.578206    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:15.578810 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:15.578788    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:15.584503 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:15.584059    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:15.584503 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:15.584192    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:16.585471 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:16.585424    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:16.585986 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:16.585581    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:16.586110 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:16.586089    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:16.586300 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:16.586202    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:16.674342 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:16.674217    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:17.584341 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:17.584300    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:17.584550 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:17.584469    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:18.207823 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.207777    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.207855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.207909    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.207980    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208002    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208008    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208023    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.207979    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208067    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.208047226 +0000 UTC m=+18.162361120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208087    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.208077006 +0000 UTC m=+18.162390891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:18.208392 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.208105    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.208094463 +0000 UTC m=+18.162408353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:18.585464 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.585378    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:18.585623 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.585479    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:18.585623 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.585511    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:18.585719 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:18.585624    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:18.600319 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.600274    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:18.600531 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.600416    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:18.600834 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:18.600744    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wzpsj"
Jul 31 14:48:19.584081 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:19.584047    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:19.584631 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:19.584202    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:20.590646 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:20.590615    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:20.591160 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:20.590639    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:20.591160 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:20.590761    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:20.591160 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:20.590909    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:21.583900 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:21.583860    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:21.584097 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:21.583980    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:22.587757 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:22.587672    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:22.588183 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:22.587797    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:22.588183 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:22.587863    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:22.588183 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:22.587925    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:23.584716 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:23.584679    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:23.584902 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:23.584819    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:24.583953 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:24.583913    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:24.584431 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:24.584054    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:24.584431 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:24.584144    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:24.584431 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:24.584268    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:25.584035 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:25.584001    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:25.584507 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:25.584124    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:26.273724 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:26.273690    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:26.273724 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:26.273740    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:26.273904 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:26.273762    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:26.273904 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273864    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.273904 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273865    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.273904 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273876    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.273904 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273899    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.274077 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273912    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.274077 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273922    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.27390566 +0000 UTC m=+34.228219547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.274077 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273935    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.273929898 +0000 UTC m=+34.228243782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.274077 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.273951    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:48:42.273940138 +0000 UTC m=+34.228254022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.584941 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:26.584466    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:26.584941 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.584598    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:26.584941 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:26.584642    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:26.584941 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:26.584742    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:27.584091 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:27.584057    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:27.584320 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:27.584190    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:28.584848 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:28.584814    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:28.585341 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:28.584923    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:28.585341 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:28.584976    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:28.585341 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:28.585066    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:29.584283 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:29.584245    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:29.584466 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:29.584372    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:30.586558 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.586318    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:30.587026 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:30.586607    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:30.587663 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.587631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:30.587773 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:30.587752    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:30.676351 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.676323    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:30.703945 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.703911    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rc6lj" event={"ID":"d003d921-8080-41f5-a0fa-65086d2e7f14","Type":"ContainerStarted","Data":"6a580acdb3c1a77bf63a352158e364ddd8075bbc91fc3eafeb3866f92f46f34a"}
Jul 31 14:48:30.709518 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.709481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"526612313a3f1fe53d9c29e17535b8ac0c6b8cc0c93032784bb9fc2ff08c7811"}
Jul 31 14:48:30.710993 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.710966    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mh4z7" event={"ID":"e130e647-10ab-4777-a1ff-a3c083c60005","Type":"ContainerStarted","Data":"6991460efa07478f722741451c175655ccaac5d873aacf9347ec0587c2e5dd2a"}
Jul 31 14:48:30.712880 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.712854    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-llt4j" event={"ID":"f4556428-410f-4ec3-90ea-2257ae552b7a","Type":"ContainerStarted","Data":"a2e98aeb1ead6deae6efaa9f39bba8df222ee43e46225e71b5f8802a4651d2a1"}
Jul 31 14:48:30.714756 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.714736    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerStarted","Data":"2483f03b3c40ad6bf85f0975954aa4aebaaf197ea399d129fb4d0a3a657ab656"}
Jul 31 14:48:30.716378 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.716318    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zp5fn" event={"ID":"7bdf2867-f4e8-43a2-b790-61b9b5591436","Type":"ContainerStarted","Data":"28f43ae263e4eb382b8bddde4525753b02e8b02749d33030ad540caa6af67364"}
Jul 31 14:48:30.718158 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.718136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4" event={"ID":"3ad1e27e-0327-47e2-8dad-8416e7d99494","Type":"ContainerStarted","Data":"6f33a81d0495adf56774262ec0f20d963c782040328c5dba081c922f5212d5c9"}
Jul 31 14:48:30.719781 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.719743    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rc6lj" podStartSLOduration=3.564134744 podStartE2EDuration="21.719730667s" totalImagesPullingTime="18.155595923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.71936266 +0000 UTC m=+22.673676568" watchObservedRunningTime="2026-07-31 14:48:30.719730667 +0000 UTC m=+22.674044573"
Jul 31 14:48:30.771184 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.771138    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-llt4j" podStartSLOduration=3.559942309 podStartE2EDuration="21.771122227s" totalImagesPullingTime="18.211179918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.771103655 +0000 UTC m=+22.725417563" watchObservedRunningTime="2026-07-31 14:48:30.771122227 +0000 UTC m=+22.725436132"
Jul 31 14:48:30.771362 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.771341    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zp5fn" podStartSLOduration=2.6099747989999997 podStartE2EDuration="21.771332391s" totalImagesPullingTime="19.161357592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.742603625 +0000 UTC m=+22.696917534" watchObservedRunningTime="2026-07-31 14:48:30.771332391 +0000 UTC m=+22.725646310"
Jul 31 14:48:30.791747 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:30.791705    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mh4z7" podStartSLOduration=3.585384275 podStartE2EDuration="21.79169s" totalImagesPullingTime="18.206305725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:30.791430315 +0000 UTC m=+22.745744246" watchObservedRunningTime="2026-07-31 14:48:30.79169 +0000 UTC m=+22.746003906"
Jul 31 14:48:31.503647 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.503297    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:30.676346482Z","UUID":"74be0c4c-31cb-4f9b-b131-009623873f59","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:31.505643 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.505616    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:31.505795 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.505663    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:31.583854 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.583825    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:31.584027 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:31.583964    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:31.722416 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.722329    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="2483f03b3c40ad6bf85f0975954aa4aebaaf197ea399d129fb4d0a3a657ab656" exitCode=0
Jul 31 14:48:31.723103 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.722443    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"2483f03b3c40ad6bf85f0975954aa4aebaaf197ea399d129fb4d0a3a657ab656"}
Jul 31 14:48:31.725662 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.725637    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:48:31.726022 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.725999    2568 generic.go:356] "Generic (PLEG): container finished" podID="d52edadf-6b08-4eda-8a43-c5d889d953fe" containerID="c8bd04a6170bb97df7830689a71e8551d922bb196cfb49e49224a367ca6caea3" exitCode=1
Jul 31 14:48:31.726117 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.726089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"c558cf4fe56ff8c94d3c3f7dcb0e39a5b96a458fff80b6eb0c50e946bba18fd2"}
Jul 31 14:48:31.726219 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.726181    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"1fcc709a9f0e388c49c5019983e24090fd1a521b0f03b3989bdd71a3e5dbd343"}
Jul 31 14:48:31.726286 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.726252    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"730694fd2e92a9c2696d1728177a428ede33db3f613e5e1c916fa094b5bf37b1"}
Jul 31 14:48:31.726342 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.726300    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"fc88b2d4e9f47dda3b6a5291ec9e203b76fb15cd7c36d95b9edb0e430c0e0b12"}
Jul 31 14:48:31.726342 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:31.726333    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerDied","Data":"c8bd04a6170bb97df7830689a71e8551d922bb196cfb49e49224a367ca6caea3"}
Jul 31 14:48:32.583929 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:32.583845    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:32.583929 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:32.583887    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:32.584118 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:32.583991    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:32.584160 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:32.584124    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:32.731467 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:32.731433    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4" event={"ID":"3ad1e27e-0327-47e2-8dad-8416e7d99494","Type":"ContainerStarted","Data":"050e7dd9ffce8b22b957f0b67e5612f3049fb1d4a0b3089d7ba399c02c26d486"}
Jul 31 14:48:32.733186 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:32.733152    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hmd2" event={"ID":"b810a9e0-78e8-41f9-9241-c49de4bdb5b5","Type":"ContainerStarted","Data":"a3d1ac54c5ae17fe4904334e68474df6608251c9d6778a6156220041e86ddaf7"}
Jul 31 14:48:32.755395 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:32.755343    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6x2d4" podStartSLOduration=3.334628064 podStartE2EDuration="23.755323664s" totalImagesPullingTime="20.4206956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:32.754722261 +0000 UTC m=+24.709036167" watchObservedRunningTime="2026-07-31 14:48:32.755323664 +0000 UTC m=+24.709637570"
Jul 31 14:48:33.584630 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:33.584558    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:33.584762 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:33.584677    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:34.584186 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:34.584148    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:34.584611 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:34.584298    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:34.584611 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:34.584353    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:34.584611 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:34.584451    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:34.745775 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:34.745555    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:48:34.746372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:34.746336    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"acb264b5cd213952888f7118a6510c1c7326512f8c79361493bd35d86a2a3b1e"}
Jul 31 14:48:35.584278 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:35.584240    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:35.584692 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:35.584353    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:35.749716 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:35.749681    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="db0bb0c11a5778b09b5cce188ba223b95a30b98509fe61d68ba537712cbfd253" exitCode=0
Jul 31 14:48:35.749716 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:35.749718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"db0bb0c11a5778b09b5cce188ba223b95a30b98509fe61d68ba537712cbfd253"}
Jul 31 14:48:35.774103 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:35.774055    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4hmd2" podStartSLOduration=8.559318976 podStartE2EDuration="26.774039868s" totalImagesPullingTime="18.214720892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:32.773343053 +0000 UTC m=+24.727656964" watchObservedRunningTime="2026-07-31 14:48:35.774039868 +0000 UTC m=+27.728353773"
Jul 31 14:48:36.584082 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:36.583998    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:36.584271 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:36.584008    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:36.584271 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:36.584097    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:36.584271 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:36.584185    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:37.584489 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.584307    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:37.584979 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:37.584570    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:37.756420 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.756306    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="befb414caa63d58513a4c2d44561f69559ffde2ce7224b34725eddf21fb29f45" exitCode=0
Jul 31 14:48:37.756420 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.756384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"befb414caa63d58513a4c2d44561f69559ffde2ce7224b34725eddf21fb29f45"}
Jul 31 14:48:37.759689 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.759670    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:48:37.760041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.760023    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"92c96353e6045766ac58d9a1eee5753e0b4a49aa4ef8abed17b9a6a5cc866c58"}
Jul 31 14:48:37.760306 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.760288    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:37.760397 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.760352    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:37.760453 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.760400    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:37.760499 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.760487    2568 scope.go:117] "RemoveContainer" containerID="c8bd04a6170bb97df7830689a71e8551d922bb196cfb49e49224a367ca6caea3"
Jul 31 14:48:37.776822 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.776803    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:37.784157 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:37.784136    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:48:38.585324 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.585297    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:38.585672 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:38.585379    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:38.585672 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.585432    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:38.585672 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:38.585508    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:38.766641 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.766570    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:48:38.766995 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.766964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" event={"ID":"d52edadf-6b08-4eda-8a43-c5d889d953fe","Type":"ContainerStarted","Data":"ffcd8b26a194ced15147746e3fd08e27addb8e136899fbd2388fadfc5474d072"}
Jul 31 14:48:38.769522 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.769498    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="18b05ef19c2d9da15d71a38cbdc525046c4e009638059883f3614c653f626534" exitCode=0
Jul 31 14:48:38.769616 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.769545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"18b05ef19c2d9da15d71a38cbdc525046c4e009638059883f3614c653f626534"}
Jul 31 14:48:38.842605 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:38.842535    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt" podStartSLOduration=11.361583929 podStartE2EDuration="29.842516997s" totalImagesPullingTime="18.480933068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:38.808492344 +0000 UTC m=+30.762806257" watchObservedRunningTime="2026-07-31 14:48:38.842516997 +0000 UTC m=+30.796830922"
Jul 31 14:48:39.191757 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.191719    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkpvs"]
Jul 31 14:48:39.191923 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.191913    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:39.192061 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:39.192037    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:39.200779 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.200747    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kbpzc"]
Jul 31 14:48:39.200926 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.200873    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:39.200994 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:39.200970    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:39.205005 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.204975    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zqnxm"]
Jul 31 14:48:39.205165 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:39.205096    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:39.205297 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:39.205267    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:40.584845 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:40.584808    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:40.584845 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:40.584828    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:40.585419 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:40.584816    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:40.585419 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:40.584945    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:40.585419 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:40.584980    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:40.585419 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:40.585050    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:42.287447 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.287410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.287471    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.287517    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287615    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287641    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287655    2568 projected.go:196] Error preparing data for projected volume kube-api-access-lncp2 for pod openshift-network-diagnostics/network-check-target-zqnxm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287666    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287719    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2 podName:2961b89c-a482-4eb0-832a-b8b31443d0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.287699424 +0000 UTC m=+66.242013333 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lncp2" (UniqueName: "kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2") pod "network-check-target-zqnxm" (UID: "2961b89c-a482-4eb0-832a-b8b31443d0ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287738    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret podName:f0ab4de6-e675-43b2-9930-5c43ad601b61 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.287731388 +0000 UTC m=+66.242045274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret") pod "global-pull-secret-syncer-kbpzc" (UID: "f0ab4de6-e675-43b2-9930-5c43ad601b61") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287753    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:42.287942 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.287788    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs podName:9680aba1-e848-493d-929a-02fa6fc70f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.287773462 +0000 UTC m=+66.242087367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs") pod "network-metrics-daemon-lkpvs" (UID: "9680aba1-e848-493d-929a-02fa6fc70f42") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:42.584747 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.584655    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:42.584747 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.584722    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:42.584993 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:42.584775    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:42.584993 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.584881    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkpvs" podUID="9680aba1-e848-493d-929a-02fa6fc70f42"
Jul 31 14:48:42.584993 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.584962    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kbpzc" podUID="f0ab4de6-e675-43b2-9930-5c43ad601b61"
Jul 31 14:48:42.585155 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:48:42.585055    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zqnxm" podUID="2961b89c-a482-4eb0-832a-b8b31443d0ed"
Jul 31 14:48:43.295142 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:43.294946    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mh4z7_e130e647-10ab-4777-a1ff-a3c083c60005/dns-node-resolver/0.log"
Jul 31 14:48:43.363040 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:43.363019    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-216.ec2.internal" event="NodeReady"
Jul 31 14:48:43.363181 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:43.363169    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:43.784639 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:43.784609    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="858eb5ff43b42a89226bd44f5b0bc9182ec032170cc6d9cfd3c0d80c21b29b63" exitCode=0
Jul 31 14:48:43.784851 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:43.784644    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"858eb5ff43b42a89226bd44f5b0bc9182ec032170cc6d9cfd3c0d80c21b29b63"}
Jul 31 14:48:44.457041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.457012    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rc6lj_d003d921-8080-41f5-a0fa-65086d2e7f14/node-ca/0.log"
Jul 31 14:48:44.584840 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.584805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:48:44.585005 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.584805    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:48:44.585072 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.584815    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:48:44.589856 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.589833    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:48:44.591011 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.590866    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:48:44.591011 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.590944    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:48:44.591011 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.590944    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:48:44.591011 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.590965    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ljf7\""
Jul 31 14:48:44.591366 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.590965    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-262fz\""
Jul 31 14:48:44.789985 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.789901    2568 generic.go:356] "Generic (PLEG): container finished" podID="0888bf70-75f8-4ecb-8d07-db4ddceb29c2" containerID="cd0291262b17eb6d906ff1f6bfd33020b22d0f7c2dc1173fc3d66ad8b5f6dd19" exitCode=0
Jul 31 14:48:44.789985 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:44.789964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerDied","Data":"cd0291262b17eb6d906ff1f6bfd33020b22d0f7c2dc1173fc3d66ad8b5f6dd19"}
Jul 31 14:48:45.795660 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:45.795625    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" event={"ID":"0888bf70-75f8-4ecb-8d07-db4ddceb29c2","Type":"ContainerStarted","Data":"0ea824b299f010186ae078dae044c6c80e47d3fb6ac769ce3b0559b700655138"}
Jul 31 14:48:45.824412 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:48:45.824369    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ltrl7" podStartSLOduration=7.992892106 podStartE2EDuration="36.824354098s" totalImagesPullingTime="28.831461992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:45.823220328 +0000 UTC m=+37.777534226" watchObservedRunningTime="2026-07-31 14:48:45.824354098 +0000 UTC m=+37.778668004"
Jul 31 14:49:03.837055 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.837023    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"]
Jul 31 14:49:03.838833 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.838811    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:03.839470 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.839440    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"]
Jul 31 14:49:03.841238 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.841221    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"]
Jul 31 14:49:03.841299 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.841272    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.843933 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.843908    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:03.845714 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.845688    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 14:49:03.845826 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.845778    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 14:49:03.845909 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.845851    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 14:49:03.845909 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.845851    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 14:49:03.846030 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.846017    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 14:49:03.846148 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.846130    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 14:49:03.846372 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.846352    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 14:49:03.846501 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.846486    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 14:49:03.846574 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.846554    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vt76h\""
Jul 31 14:49:03.849360 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.849151    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 31 14:49:03.860333 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.860308    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"]
Jul 31 14:49:03.860836 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.860820    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"]
Jul 31 14:49:03.879079 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.879053    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"]
Jul 31 14:49:03.912128 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.912097    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9cn8d"]
Jul 31 14:49:03.914046 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.914024    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:03.917039 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.916983    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:49:03.917165 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.917148    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:49:03.917293 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.917276    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4w6d9\""
Jul 31 14:49:03.921586 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.921567    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bglfl"]
Jul 31 14:49:03.923556 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.923541    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:03.925854 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.925839    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:49:03.926135 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.926124    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:49:03.926422 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.926409    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:49:03.930648 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.930628    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lxkp8\""
Jul 31 14:49:03.930744 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.930627    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:49:03.931879 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.931862    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cn8d"]
Jul 31 14:49:03.936020 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.935996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.936116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936025    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba502ef6-d9c8-499d-b33a-295570392f75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.936116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.936116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936084    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.936116 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkx5m\" (UniqueName: \"kubernetes.io/projected/ba502ef6-d9c8-499d-b33a-295570392f75-kube-api-access-qkx5m\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.936279 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936144    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4fb\" (UniqueName: \"kubernetes.io/projected/75d0cc83-a842-4571-86ed-99af9f17bee4-kube-api-access-kf4fb\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:03.936279 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d0cc83-a842-4571-86ed-99af9f17bee4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:03.936346 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.936277    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:03.938290 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:03.938261    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bglfl"]
Jul 31 14:49:04.014269 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.014237    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9httq"]
Jul 31 14:49:04.016190 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.016175    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.019844 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.019824    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:49:04.019993 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.019923    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ls75f\""
Jul 31 14:49:04.020220 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.020185    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:49:04.020527 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.020513    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:49:04.037287 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037258    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4fb\" (UniqueName: \"kubernetes.io/projected/75d0cc83-a842-4571-86ed-99af9f17bee4-kube-api-access-kf4fb\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:04.037401 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037299    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d0cc83-a842-4571-86ed-99af9f17bee4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:04.037401 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.037515 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/debb2a9b-0bae-4fef-9de3-8620374ed8f1-data-volume\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.037515 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.037605 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037527    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0450341c-eaa9-4897-b8a9-e17b642aa99d-tmp\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.037605 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmm5\" (UniqueName: \"kubernetes.io/projected/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-api-access-zcmm5\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.037691 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037648    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba502ef6-d9c8-499d-b33a-295570392f75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.037691 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.037783 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037701    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddbb76d-dfbc-41aa-b6ca-41389433dafd-tmp-dir\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.037783 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037731    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7h6g\" (UniqueName: \"kubernetes.io/projected/bddbb76d-dfbc-41aa-b6ca-41389433dafd-kube-api-access-h7h6g\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.037886 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037849    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0450341c-eaa9-4897-b8a9-e17b642aa99d-klusterlet-config\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.037944 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037885    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.037944 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037933    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kck2\" (UniqueName: \"kubernetes.io/projected/0450341c-eaa9-4897-b8a9-e17b642aa99d-kube-api-access-6kck2\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.038041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037964    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbb76d-dfbc-41aa-b6ca-41389433dafd-config-volume\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.038041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.037993    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.038041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.038025    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkx5m\" (UniqueName: \"kubernetes.io/projected/ba502ef6-d9c8-499d-b33a-295570392f75-kube-api-access-qkx5m\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.038188 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.038051    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bddbb76d-dfbc-41aa-b6ca-41389433dafd-metrics-tls\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.038188 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.038101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/debb2a9b-0bae-4fef-9de3-8620374ed8f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.038188 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.038128    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/debb2a9b-0bae-4fef-9de3-8620374ed8f1-crio-socket\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.038508 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.038489    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba502ef6-d9c8-499d-b33a-295570392f75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.041928 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.041880    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-ca\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.042032 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.041983    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.042131 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.042109    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.042131 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.042123    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ba502ef6-d9c8-499d-b33a-295570392f75-hub\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.042515 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.042496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75d0cc83-a842-4571-86ed-99af9f17bee4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:04.042582 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.042567    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9httq"]
Jul 31 14:49:04.059352 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.059328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkx5m\" (UniqueName: \"kubernetes.io/projected/ba502ef6-d9c8-499d-b33a-295570392f75-kube-api-access-qkx5m\") pod \"cluster-proxy-proxy-agent-5b49c5d494-l7jsc\" (UID: \"ba502ef6-d9c8-499d-b33a-295570392f75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.059467 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.059416    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4fb\" (UniqueName: \"kubernetes.io/projected/75d0cc83-a842-4571-86ed-99af9f17bee4-kube-api-access-kf4fb\") pod \"managed-serviceaccount-addon-agent-7cc875dc7d-q9njd\" (UID: \"75d0cc83-a842-4571-86ed-99af9f17bee4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:04.139478 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139394    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0450341c-eaa9-4897-b8a9-e17b642aa99d-tmp\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.139478 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139430    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmm5\" (UniqueName: \"kubernetes.io/projected/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-api-access-zcmm5\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.139478 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139453    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddbb76d-dfbc-41aa-b6ca-41389433dafd-tmp-dir\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.139478 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139472    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7h6g\" (UniqueName: \"kubernetes.io/projected/bddbb76d-dfbc-41aa-b6ca-41389433dafd-kube-api-access-h7h6g\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139595    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0450341c-eaa9-4897-b8a9-e17b642aa99d-klusterlet-config\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139677    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kck2\" (UniqueName: \"kubernetes.io/projected/0450341c-eaa9-4897-b8a9-e17b642aa99d-kube-api-access-6kck2\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139707    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbb76d-dfbc-41aa-b6ca-41389433dafd-config-volume\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139737    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dct5\" (UniqueName: \"kubernetes.io/projected/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-kube-api-access-7dct5\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.139807 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139769    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bddbb76d-dfbc-41aa-b6ca-41389433dafd-metrics-tls\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.140031 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139779    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddbb76d-dfbc-41aa-b6ca-41389433dafd-tmp-dir\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.140099 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.139824    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-cert\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.140158 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140139    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/debb2a9b-0bae-4fef-9de3-8620374ed8f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.140233 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140175    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0450341c-eaa9-4897-b8a9-e17b642aa99d-tmp\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.140306 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/debb2a9b-0bae-4fef-9de3-8620374ed8f1-crio-socket\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.140419 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140357    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/debb2a9b-0bae-4fef-9de3-8620374ed8f1-crio-socket\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.140824 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140527    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bddbb76d-dfbc-41aa-b6ca-41389433dafd-config-volume\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.140824 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140563    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/debb2a9b-0bae-4fef-9de3-8620374ed8f1-data-volume\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.140824 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140784    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.141035 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.140840    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/debb2a9b-0bae-4fef-9de3-8620374ed8f1-data-volume\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.143111 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.143078    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0450341c-eaa9-4897-b8a9-e17b642aa99d-klusterlet-config\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.143111 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.143100    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bddbb76d-dfbc-41aa-b6ca-41389433dafd-metrics-tls\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.145015 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.144995    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/debb2a9b-0bae-4fef-9de3-8620374ed8f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.151770 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.151750    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7h6g\" (UniqueName: \"kubernetes.io/projected/bddbb76d-dfbc-41aa-b6ca-41389433dafd-kube-api-access-h7h6g\") pod \"dns-default-9cn8d\" (UID: \"bddbb76d-dfbc-41aa-b6ca-41389433dafd\") " pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.153352 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.153331    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kck2\" (UniqueName: \"kubernetes.io/projected/0450341c-eaa9-4897-b8a9-e17b642aa99d-kube-api-access-6kck2\") pod \"klusterlet-addon-workmgr-bdf67cc7b-dmglf\" (UID: \"0450341c-eaa9-4897-b8a9-e17b642aa99d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.153636 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.153620    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmm5\" (UniqueName: \"kubernetes.io/projected/debb2a9b-0bae-4fef-9de3-8620374ed8f1-kube-api-access-zcmm5\") pod \"insights-runtime-extractor-bglfl\" (UID: \"debb2a9b-0bae-4fef-9de3-8620374ed8f1\") " pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.161555 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.161541    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"
Jul 31 14:49:04.168987 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.168964    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:04.171870 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.171851    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"
Jul 31 14:49:04.245577 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.245540    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dct5\" (UniqueName: \"kubernetes.io/projected/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-kube-api-access-7dct5\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.245737 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.245619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-cert\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.246355 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.246329    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:04.250359 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.250096    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bglfl"
Jul 31 14:49:04.252404 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.251931    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-cert\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.259539 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.259328    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dct5\" (UniqueName: \"kubernetes.io/projected/bb6c2d69-bf91-4966-b851-e84b6f31f4c7-kube-api-access-7dct5\") pod \"ingress-canary-9httq\" (UID: \"bb6c2d69-bf91-4966-b851-e84b6f31f4c7\") " pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.324571 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.324532    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9httq"
Jul 31 14:49:04.355991 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.355957    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"]
Jul 31 14:49:04.362035 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.361084    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0450341c_eaa9_4897_b8a9_e17b642aa99d.slice/crio-209542dec7b870f05b3e6c2e03087c283050f0a9c24ae9304b70181d63032f73 WatchSource:0}: Error finding container 209542dec7b870f05b3e6c2e03087c283050f0a9c24ae9304b70181d63032f73: Status 404 returned error can't find the container with id 209542dec7b870f05b3e6c2e03087c283050f0a9c24ae9304b70181d63032f73
Jul 31 14:49:04.380466 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.380328    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd"]
Jul 31 14:49:04.384674 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.384617    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d0cc83_a842_4571_86ed_99af9f17bee4.slice/crio-57b9e62c1605717df6b500fa93a919c712766ea4e5e4c0968b8849a1398f119f WatchSource:0}: Error finding container 57b9e62c1605717df6b500fa93a919c712766ea4e5e4c0968b8849a1398f119f: Status 404 returned error can't find the container with id 57b9e62c1605717df6b500fa93a919c712766ea4e5e4c0968b8849a1398f119f
Jul 31 14:49:04.396235 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.396189    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc"]
Jul 31 14:49:04.403594 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.403559    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba502ef6_d9c8_499d_b33a_295570392f75.slice/crio-aed88ac976ffc2273123236ec3be5d6c27fec8d866bffb3ea493ce760ad2719a WatchSource:0}: Error finding container aed88ac976ffc2273123236ec3be5d6c27fec8d866bffb3ea493ce760ad2719a: Status 404 returned error can't find the container with id aed88ac976ffc2273123236ec3be5d6c27fec8d866bffb3ea493ce760ad2719a
Jul 31 14:49:04.449295 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.449263    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bglfl"]
Jul 31 14:49:04.453365 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.453327    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebb2a9b_0bae_4fef_9de3_8620374ed8f1.slice/crio-e7bb17aa78c400075686f0492af518b72476bec7c79c2eb04a3c0bb645715df2 WatchSource:0}: Error finding container e7bb17aa78c400075686f0492af518b72476bec7c79c2eb04a3c0bb645715df2: Status 404 returned error can't find the container with id e7bb17aa78c400075686f0492af518b72476bec7c79c2eb04a3c0bb645715df2
Jul 31 14:49:04.471608 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.471587    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9cn8d"]
Jul 31 14:49:04.486339 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.486306    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddbb76d_dfbc_41aa_b6ca_41389433dafd.slice/crio-698321138b717e86659cecfedafdb0092d80c330461b6dfc725b96ecc2777a7f WatchSource:0}: Error finding container 698321138b717e86659cecfedafdb0092d80c330461b6dfc725b96ecc2777a7f: Status 404 returned error can't find the container with id 698321138b717e86659cecfedafdb0092d80c330461b6dfc725b96ecc2777a7f
Jul 31 14:49:04.515428 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.515399    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9httq"]
Jul 31 14:49:04.518374 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:04.518340    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6c2d69_bf91_4966_b851_e84b6f31f4c7.slice/crio-afb42067567ebcbe7bc53f4237ddcd14365f857e1add1f9bee06cb7935b529cc WatchSource:0}: Error finding container afb42067567ebcbe7bc53f4237ddcd14365f857e1add1f9bee06cb7935b529cc: Status 404 returned error can't find the container with id afb42067567ebcbe7bc53f4237ddcd14365f857e1add1f9bee06cb7935b529cc
Jul 31 14:49:04.841076 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.841045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cn8d" event={"ID":"bddbb76d-dfbc-41aa-b6ca-41389433dafd","Type":"ContainerStarted","Data":"698321138b717e86659cecfedafdb0092d80c330461b6dfc725b96ecc2777a7f"}
Jul 31 14:49:04.842241 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.842215    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc" event={"ID":"ba502ef6-d9c8-499d-b33a-295570392f75","Type":"ContainerStarted","Data":"aed88ac976ffc2273123236ec3be5d6c27fec8d866bffb3ea493ce760ad2719a"}
Jul 31 14:49:04.843246 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.843221    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf" event={"ID":"0450341c-eaa9-4897-b8a9-e17b642aa99d","Type":"ContainerStarted","Data":"209542dec7b870f05b3e6c2e03087c283050f0a9c24ae9304b70181d63032f73"}
Jul 31 14:49:04.844306 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.844286    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd" event={"ID":"75d0cc83-a842-4571-86ed-99af9f17bee4","Type":"ContainerStarted","Data":"57b9e62c1605717df6b500fa93a919c712766ea4e5e4c0968b8849a1398f119f"}
Jul 31 14:49:04.845670 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.845649    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bglfl" event={"ID":"debb2a9b-0bae-4fef-9de3-8620374ed8f1","Type":"ContainerStarted","Data":"24b9bf056dc00495fa3dc1a01a9d7b3a4d0f6e191972edc675bb0f3c349f0ab6"}
Jul 31 14:49:04.845762 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.845693    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bglfl" event={"ID":"debb2a9b-0bae-4fef-9de3-8620374ed8f1","Type":"ContainerStarted","Data":"e7bb17aa78c400075686f0492af518b72476bec7c79c2eb04a3c0bb645715df2"}
Jul 31 14:49:04.846612 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:04.846591    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9httq" event={"ID":"bb6c2d69-bf91-4966-b851-e84b6f31f4c7","Type":"ContainerStarted","Data":"afb42067567ebcbe7bc53f4237ddcd14365f857e1add1f9bee06cb7935b529cc"}
Jul 31 14:49:05.860403 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:05.859893    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bglfl" event={"ID":"debb2a9b-0bae-4fef-9de3-8620374ed8f1","Type":"ContainerStarted","Data":"6d232675cb3a5c994900b4033df71ed2a610b272c0dd7cffa4b53e36a1639d8a"}
Jul 31 14:49:09.784302 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:09.784267    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b95kt"
Jul 31 14:49:13.888378 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.888336    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd" event={"ID":"75d0cc83-a842-4571-86ed-99af9f17bee4","Type":"ContainerStarted","Data":"2f43171da9a77654c87e5e775dde79692e986f0e03e25fcc50bf4daa601701b8"}
Jul 31 14:49:13.890646 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.890613    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bglfl" event={"ID":"debb2a9b-0bae-4fef-9de3-8620374ed8f1","Type":"ContainerStarted","Data":"6da2fdae05082a8572ce63448bb58debf76ed67f42a5cc0f64c122dbca74dfba"}
Jul 31 14:49:13.891970 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.891946    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9httq" event={"ID":"bb6c2d69-bf91-4966-b851-e84b6f31f4c7","Type":"ContainerStarted","Data":"f60dfc861a0acf84090b5da3b237badb94fa50752ee5abcb73a21c52ac8a01c0"}
Jul 31 14:49:13.893537 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.893517    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cn8d" event={"ID":"bddbb76d-dfbc-41aa-b6ca-41389433dafd","Type":"ContainerStarted","Data":"e71b68ac587d7fe4ba1a3c62dda7c53802ca514065ebcca10783c6125200753f"}
Jul 31 14:49:13.893637 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.893553    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9cn8d" event={"ID":"bddbb76d-dfbc-41aa-b6ca-41389433dafd","Type":"ContainerStarted","Data":"53ac0aafa186a8094b82babc9d8501d3739ff849d886994dc8dd8550f5a81ab1"}
Jul 31 14:49:13.893637 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.893573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:13.895423 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.895402    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc" event={"ID":"ba502ef6-d9c8-499d-b33a-295570392f75","Type":"ContainerStarted","Data":"107c63a9136b339bd74171e33b10f6ae6e99c87d2ded7da8d2c1c2f0669a9c5d"}
Jul 31 14:49:13.895524 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.895437    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc" event={"ID":"ba502ef6-d9c8-499d-b33a-295570392f75","Type":"ContainerStarted","Data":"e9361c64a78113701833a289623b72bf788583d4af702be1855499c99eff3d21"}
Jul 31 14:49:13.895524 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.895456    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc" event={"ID":"ba502ef6-d9c8-499d-b33a-295570392f75","Type":"ContainerStarted","Data":"e6bd4a840a4e30e58a57610823cea81d090136d47fce180c3011a58e990497f9"}
Jul 31 14:49:13.896650 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.896633    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf" event={"ID":"0450341c-eaa9-4897-b8a9-e17b642aa99d","Type":"ContainerStarted","Data":"14ee349bcce43ddaf7eaeb47fe46ba1f87934ab35aee80ea30dac158787149b5"}
Jul 31 14:49:13.896846 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.896832    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:13.898686 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.898667    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf"
Jul 31 14:49:13.913911 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.913872    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7cc875dc7d-q9njd" podStartSLOduration=2.196817765 podStartE2EDuration="10.913861224s" totalImagesPullingTime="8.717043459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:13.911519837 +0000 UTC m=+65.865833743" watchObservedRunningTime="2026-07-31 14:49:13.913861224 +0000 UTC m=+65.868175129"
Jul 31 14:49:13.959556 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.959502    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9cn8d" podStartSLOduration=2.35776624 podStartE2EDuration="10.959481364s" totalImagesPullingTime="8.601715124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:13.95828053 +0000 UTC m=+65.912594568" watchObservedRunningTime="2026-07-31 14:49:13.959481364 +0000 UTC m=+65.913795248"
Jul 31 14:49:13.986946 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:13.986900    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bglfl" podStartSLOduration=2.57431337 podStartE2EDuration="10.986884943s" totalImagesPullingTime="8.412571573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:13.98564523 +0000 UTC m=+65.939959137" watchObservedRunningTime="2026-07-31 14:49:13.986884943 +0000 UTC m=+65.941198849"
Jul 31 14:49:14.212585 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.212530    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9httq" podStartSLOduration=2.637981778 podStartE2EDuration="11.212511765s" totalImagesPullingTime="8.574529987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:14.13471155 +0000 UTC m=+66.089025456" watchObservedRunningTime="2026-07-31 14:49:14.212511765 +0000 UTC m=+66.166825671"
Jul 31 14:49:14.260655 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.260586    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5b49c5d494-l7jsc" podStartSLOduration=2.557732584 podStartE2EDuration="11.260565689s" totalImagesPullingTime="8.702833105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:14.227259684 +0000 UTC m=+66.181573590" watchObservedRunningTime="2026-07-31 14:49:14.260565689 +0000 UTC m=+66.214879599"
Jul 31 14:49:14.261348 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.261301    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-bdf67cc7b-dmglf" podStartSLOduration=2.515077865 podStartE2EDuration="11.261285431s" totalImagesPullingTime="8.746207566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:14.25998734 +0000 UTC m=+66.214301248" watchObservedRunningTime="2026-07-31 14:49:14.261285431 +0000 UTC m=+66.215599338"
Jul 31 14:49:14.336632 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.336595    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:49:14.336831 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.336644    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:49:14.336831 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.336754    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:49:14.341419 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.341398    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:14.341490 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.341403    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:49:14.341490 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.341432    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:14.349694 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.349671    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0ab4de6-e675-43b2-9930-5c43ad601b61-original-pull-secret\") pod \"global-pull-secret-syncer-kbpzc\" (UID: \"f0ab4de6-e675-43b2-9930-5c43ad601b61\") " pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:49:14.349795 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.349705    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9680aba1-e848-493d-929a-02fa6fc70f42-metrics-certs\") pod \"network-metrics-daemon-lkpvs\" (UID: \"9680aba1-e848-493d-929a-02fa6fc70f42\") " pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:49:14.349973 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.349959    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:49:14.360579 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.360548    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lncp2\" (UniqueName: \"kubernetes.io/projected/2961b89c-a482-4eb0-832a-b8b31443d0ed-kube-api-access-lncp2\") pod \"network-check-target-zqnxm\" (UID: \"2961b89c-a482-4eb0-832a-b8b31443d0ed\") " pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:49:14.598883 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.598805    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ljf7\""
Jul 31 14:49:14.600954 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.600939    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kbpzc"
Jul 31 14:49:14.606168 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.606144    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:49:14.608991 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.608975    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-262fz\""
Jul 31 14:49:14.616129 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.616115    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkpvs"
Jul 31 14:49:14.777179 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.777010    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kbpzc"]
Jul 31 14:49:14.780813 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:14.780033    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ab4de6_e675_43b2_9930_5c43ad601b61.slice/crio-f55ea1b0bd1c63cb11363cecc2c38361e2b7615830e72c83c67f791e4408ac6d WatchSource:0}: Error finding container f55ea1b0bd1c63cb11363cecc2c38361e2b7615830e72c83c67f791e4408ac6d: Status 404 returned error can't find the container with id f55ea1b0bd1c63cb11363cecc2c38361e2b7615830e72c83c67f791e4408ac6d
Jul 31 14:49:14.799162 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.799132    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zqnxm"]
Jul 31 14:49:14.803125 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:14.803100    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2961b89c_a482_4eb0_832a_b8b31443d0ed.slice/crio-f58b1264499b92a7eceb38318f591c798c791a610b943f039be27a821043a511 WatchSource:0}: Error finding container f58b1264499b92a7eceb38318f591c798c791a610b943f039be27a821043a511: Status 404 returned error can't find the container with id f58b1264499b92a7eceb38318f591c798c791a610b943f039be27a821043a511
Jul 31 14:49:14.836697 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.836677    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkpvs"]
Jul 31 14:49:14.838897 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:14.838874    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9680aba1_e848_493d_929a_02fa6fc70f42.slice/crio-888bbf1971d0cacf6bf4ba121320869d52a7c9f126961b268a497b6fc2a3634b WatchSource:0}: Error finding container 888bbf1971d0cacf6bf4ba121320869d52a7c9f126961b268a497b6fc2a3634b: Status 404 returned error can't find the container with id 888bbf1971d0cacf6bf4ba121320869d52a7c9f126961b268a497b6fc2a3634b
Jul 31 14:49:14.900872 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.900782    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkpvs" event={"ID":"9680aba1-e848-493d-929a-02fa6fc70f42","Type":"ContainerStarted","Data":"888bbf1971d0cacf6bf4ba121320869d52a7c9f126961b268a497b6fc2a3634b"}
Jul 31 14:49:14.901925 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.901902    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kbpzc" event={"ID":"f0ab4de6-e675-43b2-9930-5c43ad601b61","Type":"ContainerStarted","Data":"f55ea1b0bd1c63cb11363cecc2c38361e2b7615830e72c83c67f791e4408ac6d"}
Jul 31 14:49:14.903001 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:14.902979    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zqnxm" event={"ID":"2961b89c-a482-4eb0-832a-b8b31443d0ed","Type":"ContainerStarted","Data":"f58b1264499b92a7eceb38318f591c798c791a610b943f039be27a821043a511"}
Jul 31 14:49:16.192899 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.192012    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2pmzv"]
Jul 31 14:49:16.194594 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.194571    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.198879 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.198859    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:16.204083 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.204060    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:49:16.205325 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.205308    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:49:16.205398 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.205336    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t7hpg\""
Jul 31 14:49:16.205572 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.205557    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:49:16.205676 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.205660    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:49:16.205761 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.205749    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:49:16.250976 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.250951    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-textfile\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251095 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.250990    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-metrics-client-ca\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251095 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251021    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251095 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251053    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251095 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251083    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-sys\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251333 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251112    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-wtmp\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251333 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251150    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkt4w\" (UniqueName: \"kubernetes.io/projected/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-kube-api-access-jkt4w\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251333 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251179    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-root\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.251472 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.251397    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352477 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352448    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352582 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352499    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-textfile\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352582 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352524    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-metrics-client-ca\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352697 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352585    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352697 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352619    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352697 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352650    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-sys\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352840 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352735    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-wtmp\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352840 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352776    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkt4w\" (UniqueName: \"kubernetes.io/projected/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-kube-api-access-jkt4w\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352840 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352810    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-root\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.352968 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.352897    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-root\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.353853    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.354159    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-textfile\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.354643    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-metrics-client-ca\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:49:16.354737    2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:49:16.354792    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls podName:8a42b7dd-571c-43d1-b36d-7d0d8b20540d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:16.854772747 +0000 UTC m=+68.809086632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls") pod "node-exporter-2pmzv" (UID: "8a42b7dd-571c-43d1-b36d-7d0d8b20540d") : secret "node-exporter-tls" not found
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.354919    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-wtmp\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.355309 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.354966    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-sys\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.358581 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.358536    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.366214 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.366164    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkt4w\" (UniqueName: \"kubernetes.io/projected/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-kube-api-access-jkt4w\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.858006 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.857531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.860624 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.860594    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a42b7dd-571c-43d1-b36d-7d0d8b20540d-node-exporter-tls\") pod \"node-exporter-2pmzv\" (UID: \"8a42b7dd-571c-43d1-b36d-7d0d8b20540d\") " pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:16.916306 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.916273    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkpvs" event={"ID":"9680aba1-e848-493d-929a-02fa6fc70f42","Type":"ContainerStarted","Data":"73ca92e1ab977ead5f4ef91fec3f29f9f10f7c4089fd0d217955a6fa4e7ca5f8"}
Jul 31 14:49:16.916480 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:16.916335    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkpvs" event={"ID":"9680aba1-e848-493d-929a-02fa6fc70f42","Type":"ContainerStarted","Data":"8472935caa2b86ff15b87337a091c1100434b7b5fe8a9a71e9992fc16de7d0ca"}
Jul 31 14:49:17.125759 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:17.125663    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2pmzv"
Jul 31 14:49:19.270067 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:49:19.270029    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a42b7dd_571c_43d1_b36d_7d0d8b20540d.slice/crio-33e4fa28eb4d5b7a51c39afb61761cfe324b38fc4609138a70dc4ca5a6472f6c WatchSource:0}: Error finding container 33e4fa28eb4d5b7a51c39afb61761cfe324b38fc4609138a70dc4ca5a6472f6c: Status 404 returned error can't find the container with id 33e4fa28eb4d5b7a51c39afb61761cfe324b38fc4609138a70dc4ca5a6472f6c
Jul 31 14:49:19.934782 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:19.934737    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zqnxm" event={"ID":"2961b89c-a482-4eb0-832a-b8b31443d0ed","Type":"ContainerStarted","Data":"b0ea7111ebbd29a2eec5df8aa8d5dc158a295a1dc5f9ed1bafa5000155ab8648"}
Jul 31 14:49:19.934979 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:19.934827    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:49:19.936611 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:19.936583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kbpzc" event={"ID":"f0ab4de6-e675-43b2-9930-5c43ad601b61","Type":"ContainerStarted","Data":"29833f2cc0cf47b274bd063247357d203cd685aa64709ca440a608f129ea1509"}
Jul 31 14:49:19.937928 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:19.937903    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pmzv" event={"ID":"8a42b7dd-571c-43d1-b36d-7d0d8b20540d","Type":"ContainerStarted","Data":"33e4fa28eb4d5b7a51c39afb61761cfe324b38fc4609138a70dc4ca5a6472f6c"}
Jul 31 14:49:19.969350 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:19.969294    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lkpvs" podStartSLOduration=69.689904511 podStartE2EDuration="1m10.969273537s" totalImagesPullingTime="1.279369026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:16.952404495 +0000 UTC m=+68.906718402" watchObservedRunningTime="2026-07-31 14:49:19.969273537 +0000 UTC m=+71.923587446"
Jul 31 14:49:20.008297 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:20.008239    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zqnxm" podStartSLOduration=66.538306654 podStartE2EDuration="1m11.008188565s" totalImagesPullingTime="4.469881911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:19.982278919 +0000 UTC m=+71.936592827" watchObservedRunningTime="2026-07-31 14:49:20.008188565 +0000 UTC m=+71.962502472"
Jul 31 14:49:20.943278 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:20.943247    2568 generic.go:356] "Generic (PLEG): container finished" podID="8a42b7dd-571c-43d1-b36d-7d0d8b20540d" containerID="6cbc38bd33611794fc5813372f1c56899398299fbb62114e33151ad64e648809" exitCode=0
Jul 31 14:49:20.943698 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:20.943339    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pmzv" event={"ID":"8a42b7dd-571c-43d1-b36d-7d0d8b20540d","Type":"ContainerDied","Data":"6cbc38bd33611794fc5813372f1c56899398299fbb62114e33151ad64e648809"}
Jul 31 14:49:20.978583 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:20.978528    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kbpzc" podStartSLOduration=67.479478374 podStartE2EDuration="1m11.978504032s" totalImagesPullingTime="4.499025658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:20.009258746 +0000 UTC m=+71.963572635" watchObservedRunningTime="2026-07-31 14:49:20.978504032 +0000 UTC m=+72.932817937"
Jul 31 14:49:21.949179 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:21.949130    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pmzv" event={"ID":"8a42b7dd-571c-43d1-b36d-7d0d8b20540d","Type":"ContainerStarted","Data":"b55f1ad141a9713c6c9f3c33abe3e86e8ea0a52e8e7b5502e6cbc2d5aefddd74"}
Jul 31 14:49:21.949610 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:21.949236    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2pmzv" event={"ID":"8a42b7dd-571c-43d1-b36d-7d0d8b20540d","Type":"ContainerStarted","Data":"50165151170c209785f1951aad86c92202efc4b7eb41d7b81c76f7a9d106acd8"}
Jul 31 14:49:21.974502 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:21.974444    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2pmzv" podStartSLOduration=5.054868136 podStartE2EDuration="5.974426637s" totalImagesPullingTime="849.891322ms" totalInitContainerRuntime="69.667179ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:21.973277828 +0000 UTC m=+73.927591734" watchObservedRunningTime="2026-07-31 14:49:21.974426637 +0000 UTC m=+73.928740543"
Jul 31 14:49:23.904974 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:23.904940    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9cn8d"
Jul 31 14:49:50.946832 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:49:50.946800    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zqnxm"
Jul 31 14:50:05.413007 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:50:05.412970    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9httq_bb6c2d69-bf91-4966-b851-e84b6f31f4c7/serve-healthcheck-canary/0.log"
Jul 31 14:53:08.420316 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:53:08.420291    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:53:08.426204 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:53:08.426176    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:53:08.426898 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:53:08.426878    2568 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:54:36.323649 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.323564    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-zms2q"]
Jul 31 14:54:36.325408 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.325393    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.329018 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.328997    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 14:54:36.329138 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.329123    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 14:54:36.330027 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.329999    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 31 14:54:36.330027 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.330004    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 31 14:54:36.330178 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.330024    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c9hsm\""
Jul 31 14:54:36.330178 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.330104    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 14:54:36.341622 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.341589    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zms2q"]
Jul 31 14:54:36.403563 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.403529    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xpg\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-kube-api-access-85xpg\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.403738 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.403575    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d698aefa-6d9f-4b42-9fef-734b91f1ac01-cabundle0\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.403738 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.403659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.504343 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.504307    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.504343 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.504355    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xpg\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-kube-api-access-85xpg\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.504563 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.504399    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d698aefa-6d9f-4b42-9fef-734b91f1ac01-cabundle0\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.504563 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:36.504476    2568 secret.go:281] references non-existent secret key: ca.crt
Jul 31 14:54:36.504563 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:36.504494    2568 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 14:54:36.504563 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:36.504504    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zms2q: references non-existent secret key: ca.crt
Jul 31 14:54:36.504704 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:36.504585    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates podName:d698aefa-6d9f-4b42-9fef-734b91f1ac01 nodeName:}" failed. No retries permitted until 2026-07-31 14:54:37.00456455 +0000 UTC m=+388.958878439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates") pod "keda-operator-77f648d694-zms2q" (UID: "d698aefa-6d9f-4b42-9fef-734b91f1ac01") : references non-existent secret key: ca.crt
Jul 31 14:54:36.505053 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.505036    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d698aefa-6d9f-4b42-9fef-734b91f1ac01-cabundle0\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:36.523570 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:36.523547    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xpg\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-kube-api-access-85xpg\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:37.009106 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.009070    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:37.009309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.009178    2568 secret.go:281] references non-existent secret key: ca.crt
Jul 31 14:54:37.009309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.009219    2568 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 31 14:54:37.009309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.009229    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zms2q: references non-existent secret key: ca.crt
Jul 31 14:54:37.009309 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.009286    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates podName:d698aefa-6d9f-4b42-9fef-734b91f1ac01 nodeName:}" failed. No retries permitted until 2026-07-31 14:54:38.009271519 +0000 UTC m=+389.963585403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates") pod "keda-operator-77f648d694-zms2q" (UID: "d698aefa-6d9f-4b42-9fef-734b91f1ac01") : references non-existent secret key: ca.crt
Jul 31 14:54:37.339692 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.339606    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-hwzz7"]
Jul 31 14:54:37.341638 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.341620    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:37.346425 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.346404    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 31 14:54:37.369176 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.369148    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-hwzz7"]
Jul 31 14:54:37.411946 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.411912    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tmz\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-kube-api-access-d5tmz\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:37.412109 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.411989    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:37.512736 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.512695    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:37.512940 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.512754    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tmz\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-kube-api-access-d5tmz\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:37.512940 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.512866    2568 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 31 14:54:37.512940 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.512898    2568 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-hwzz7: secret "keda-admission-webhooks-certs" not found
Jul 31 14:54:37.513114 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:54:37.512953    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates podName:e21d1428-8660-49df-84d3-ffb8f5f8f130 nodeName:}" failed. No retries permitted until 2026-07-31 14:54:38.012933958 +0000 UTC m=+389.967247857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates") pod "keda-admission-56f9798bb4-hwzz7" (UID: "e21d1428-8660-49df-84d3-ffb8f5f8f130") : secret "keda-admission-webhooks-certs" not found
Jul 31 14:54:37.544507 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:37.544472    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tmz\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-kube-api-access-d5tmz\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:38.017217 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.017167    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:38.017412 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.017235    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:38.019814 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.019792    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d698aefa-6d9f-4b42-9fef-734b91f1ac01-certificates\") pod \"keda-operator-77f648d694-zms2q\" (UID: \"d698aefa-6d9f-4b42-9fef-734b91f1ac01\") " pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:38.019935 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.019870    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e21d1428-8660-49df-84d3-ffb8f5f8f130-certificates\") pod \"keda-admission-56f9798bb4-hwzz7\" (UID: \"e21d1428-8660-49df-84d3-ffb8f5f8f130\") " pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:38.135738 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.135703    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:38.251795 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.251758    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:38.349813 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.349782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zms2q"]
Jul 31 14:54:38.356412 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:54:38.356380    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd698aefa_6d9f_4b42_9fef_734b91f1ac01.slice/crio-19510cd468d03d9d1362969302ca327f55cdf97a9bbc03d1182adc68596277fe WatchSource:0}: Error finding container 19510cd468d03d9d1362969302ca327f55cdf97a9bbc03d1182adc68596277fe: Status 404 returned error can't find the container with id 19510cd468d03d9d1362969302ca327f55cdf97a9bbc03d1182adc68596277fe
Jul 31 14:54:38.357952 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.357935    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:54:38.408404 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.408377    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-hwzz7"]
Jul 31 14:54:38.410674 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:54:38.410650    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21d1428_8660_49df_84d3_ffb8f5f8f130.slice/crio-a3e4cc90a0ad105f4e68b48f823f4139ec5c14344ca5addb951970bcc20f7e74 WatchSource:0}: Error finding container a3e4cc90a0ad105f4e68b48f823f4139ec5c14344ca5addb951970bcc20f7e74: Status 404 returned error can't find the container with id a3e4cc90a0ad105f4e68b48f823f4139ec5c14344ca5addb951970bcc20f7e74
Jul 31 14:54:38.956741 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.956687    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7" event={"ID":"e21d1428-8660-49df-84d3-ffb8f5f8f130","Type":"ContainerStarted","Data":"a3e4cc90a0ad105f4e68b48f823f4139ec5c14344ca5addb951970bcc20f7e74"}
Jul 31 14:54:38.957644 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:38.957619    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zms2q" event={"ID":"d698aefa-6d9f-4b42-9fef-734b91f1ac01","Type":"ContainerStarted","Data":"19510cd468d03d9d1362969302ca327f55cdf97a9bbc03d1182adc68596277fe"}
Jul 31 14:54:41.971821 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:41.971776    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7" event={"ID":"e21d1428-8660-49df-84d3-ffb8f5f8f130","Type":"ContainerStarted","Data":"7e997539b11752e6fd201c373a3dc1c9ba13fd5bfc719319a14ebd65708dfa92"}
Jul 31 14:54:41.972278 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:41.971866    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:54:41.994099 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:41.994040    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7" podStartSLOduration=2.114880385 podStartE2EDuration="4.994018722s" totalImagesPullingTime="2.879138337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:41.993436313 +0000 UTC m=+393.947750221" watchObservedRunningTime="2026-07-31 14:54:41.994018722 +0000 UTC m=+393.948332633"
Jul 31 14:54:42.976557 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:42.976514    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zms2q" event={"ID":"d698aefa-6d9f-4b42-9fef-734b91f1ac01","Type":"ContainerStarted","Data":"9b931f9d50deecf3186db6c97a5483d6f223b427aeead9392e6b6ac42fd637c8"}
Jul 31 14:54:42.977000 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:42.976661    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:54:42.995537 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:54:42.995481    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-zms2q" podStartSLOduration=2.9305639709999998 podStartE2EDuration="6.995463687s" totalImagesPullingTime="4.064899716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:42.994018348 +0000 UTC m=+394.948332254" watchObservedRunningTime="2026-07-31 14:54:42.995463687 +0000 UTC m=+394.949777592"
Jul 31 14:55:02.978947 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:02.978915    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-hwzz7"
Jul 31 14:55:03.981977 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:03.981950    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-zms2q"
Jul 31 14:55:59.537024 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.536989    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-q5kv8"]
Jul 31 14:55:59.539984 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.539967    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.549114 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.549090    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:55:59.549270 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.549214    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hhjb4\""
Jul 31 14:55:59.550066 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.550051    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:55:59.550149 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.550071    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 14:55:59.561793 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.561769    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q5kv8"]
Jul 31 14:55:59.581683 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.581652    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f7bf717-b357-4723-bf47-a5a4b84a42ac-data\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.581826 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.581703    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wn5s\" (UniqueName: \"kubernetes.io/projected/1f7bf717-b357-4723-bf47-a5a4b84a42ac-kube-api-access-7wn5s\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.682321 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.682288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f7bf717-b357-4723-bf47-a5a4b84a42ac-data\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.682502 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.682349    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wn5s\" (UniqueName: \"kubernetes.io/projected/1f7bf717-b357-4723-bf47-a5a4b84a42ac-kube-api-access-7wn5s\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.682679 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.682661    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f7bf717-b357-4723-bf47-a5a4b84a42ac-data\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.695285 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.695264    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wn5s\" (UniqueName: \"kubernetes.io/projected/1f7bf717-b357-4723-bf47-a5a4b84a42ac-kube-api-access-7wn5s\") pod \"seaweedfs-64568bcd49-q5kv8\" (UID: \"1f7bf717-b357-4723-bf47-a5a4b84a42ac\") " pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.848894 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.848796    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:55:59.992174 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:55:59.992145    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-q5kv8"]
Jul 31 14:55:59.995433 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:55:59.995405    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7bf717_b357_4723_bf47_a5a4b84a42ac.slice/crio-94c448c7e4ca058e77687de25e4c609b53050d7c8db2144b264af03634be256e WatchSource:0}: Error finding container 94c448c7e4ca058e77687de25e4c609b53050d7c8db2144b264af03634be256e: Status 404 returned error can't find the container with id 94c448c7e4ca058e77687de25e4c609b53050d7c8db2144b264af03634be256e
Jul 31 14:56:00.228751 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:56:00.228715    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q5kv8" event={"ID":"1f7bf717-b357-4723-bf47-a5a4b84a42ac","Type":"ContainerStarted","Data":"94c448c7e4ca058e77687de25e4c609b53050d7c8db2144b264af03634be256e"}
Jul 31 14:56:03.240214 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:56:03.240153    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-q5kv8" event={"ID":"1f7bf717-b357-4723-bf47-a5a4b84a42ac","Type":"ContainerStarted","Data":"77abbe66bceceae387cf673757f325c17f49409de8d150980010fa5e1493d119"}
Jul 31 14:56:03.240602 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:56:03.240278    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:56:03.259560 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:56:03.259509    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-q5kv8" podStartSLOduration=1.4252955379999999 podStartE2EDuration="4.259495462s" totalImagesPullingTime="2.834199924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:03.258945798 +0000 UTC m=+475.213259707" watchObservedRunningTime="2026-07-31 14:56:03.259495462 +0000 UTC m=+475.213809368"
Jul 31 14:56:09.246273 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:56:09.246243    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-q5kv8"
Jul 31 14:57:18.880216 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.880120    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-7vvtz"]
Jul 31 14:57:18.883166 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.883150    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:18.886782 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.886762    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 31 14:57:18.889267 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.889080    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-9ndtq\""
Jul 31 14:57:18.899922 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.899899    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7vvtz"]
Jul 31 14:57:18.903283 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.903261    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-ngp59"]
Jul 31 14:57:18.906625 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.906608    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:18.910331 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.910308    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 14:57:18.910595 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.910582    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 14:57:18.910925 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.910913    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zbvz8\""
Jul 31 14:57:18.920338 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.920310    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-ngp59"]
Jul 31 14:57:18.937650 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.937617    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjw4\" (UniqueName: \"kubernetes.io/projected/cdc048e3-f3f1-459a-b284-396f75448387-kube-api-access-xmjw4\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:18.937759 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.937669    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5rt\" (UniqueName: \"kubernetes.io/projected/62836109-e2e3-4959-8013-c742aa942aea-kube-api-access-zj5rt\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:18.937759 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.937723    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:18.937759 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.937757    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:18.937918 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:18.937825    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62836109-e2e3-4959-8013-c742aa942aea-tls-certs\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.038371 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.038334    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjw4\" (UniqueName: \"kubernetes.io/projected/cdc048e3-f3f1-459a-b284-396f75448387-kube-api-access-xmjw4\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.038546 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.038385    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5rt\" (UniqueName: \"kubernetes.io/projected/62836109-e2e3-4959-8013-c742aa942aea-kube-api-access-zj5rt\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.038546 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.038409    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.038546 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.038426    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.038546 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.038447    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62836109-e2e3-4959-8013-c742aa942aea-tls-certs\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.038781 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:57:19.038583    2568 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Jul 31 14:57:19.038781 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:57:19.038661    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs podName:cdc048e3-f3f1-459a-b284-396f75448387 nodeName:}" failed. No retries permitted until 2026-07-31 14:57:19.538644498 +0000 UTC m=+551.492958382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs") pod "odh-model-controller-8588677cfb-ngp59" (UID: "cdc048e3-f3f1-459a-b284-396f75448387") : secret "odh-model-controller-metrics-tls" not found
Jul 31 14:57:19.038781 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:57:19.038583    2568 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 31 14:57:19.038781 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:57:19.038726    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert podName:cdc048e3-f3f1-459a-b284-396f75448387 nodeName:}" failed. No retries permitted until 2026-07-31 14:57:19.53871379 +0000 UTC m=+551.493027678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert") pod "odh-model-controller-8588677cfb-ngp59" (UID: "cdc048e3-f3f1-459a-b284-396f75448387") : secret "odh-model-controller-webhook-cert" not found
Jul 31 14:57:19.041015 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.040997    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62836109-e2e3-4959-8013-c742aa942aea-tls-certs\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.048790 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.048765    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5rt\" (UniqueName: \"kubernetes.io/projected/62836109-e2e3-4959-8013-c742aa942aea-kube-api-access-zj5rt\") pod \"model-serving-api-7b954ffc95-7vvtz\" (UID: \"62836109-e2e3-4959-8013-c742aa942aea\") " pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.049146 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.049124    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjw4\" (UniqueName: \"kubernetes.io/projected/cdc048e3-f3f1-459a-b284-396f75448387-kube-api-access-xmjw4\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.192825 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.192795    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:19.330241 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.330214    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-7vvtz"]
Jul 31 14:57:19.332819 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:57:19.332793    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62836109_e2e3_4959_8013_c742aa942aea.slice/crio-70d07d2b7e9ae0421cc904c86195ab9e210515aed525191c51499c72a06828a1 WatchSource:0}: Error finding container 70d07d2b7e9ae0421cc904c86195ab9e210515aed525191c51499c72a06828a1: Status 404 returned error can't find the container with id 70d07d2b7e9ae0421cc904c86195ab9e210515aed525191c51499c72a06828a1
Jul 31 14:57:19.493324 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.493236    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7vvtz" event={"ID":"62836109-e2e3-4959-8013-c742aa942aea","Type":"ContainerStarted","Data":"70d07d2b7e9ae0421cc904c86195ab9e210515aed525191c51499c72a06828a1"}
Jul 31 14:57:19.542251 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.542216    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.542251 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.542249    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.544732 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.544709    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-cert\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.544848 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.544768    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc048e3-f3f1-459a-b284-396f75448387-metrics-certs\") pod \"odh-model-controller-8588677cfb-ngp59\" (UID: \"cdc048e3-f3f1-459a-b284-396f75448387\") " pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.815663 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.815578    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:19.986965 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:19.986892    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-ngp59"]
Jul 31 14:57:19.988999 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:57:19.988974    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc048e3_f3f1_459a_b284_396f75448387.slice/crio-143be7e98887b64dba92531fc938b9229b0dbe2dcbcd62b0174cf01a17428e46 WatchSource:0}: Error finding container 143be7e98887b64dba92531fc938b9229b0dbe2dcbcd62b0174cf01a17428e46: Status 404 returned error can't find the container with id 143be7e98887b64dba92531fc938b9229b0dbe2dcbcd62b0174cf01a17428e46
Jul 31 14:57:20.497546 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:20.497515    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-ngp59" event={"ID":"cdc048e3-f3f1-459a-b284-396f75448387","Type":"ContainerStarted","Data":"143be7e98887b64dba92531fc938b9229b0dbe2dcbcd62b0174cf01a17428e46"}
Jul 31 14:57:23.510466 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.510427    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-ngp59" event={"ID":"cdc048e3-f3f1-459a-b284-396f75448387","Type":"ContainerStarted","Data":"0f698be089fc1defe9fcf7470a131add2eaa37c157d61f0c4ab5d3bde370548a"}
Jul 31 14:57:23.510944 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.510601    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:23.511905 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.511879    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-7vvtz" event={"ID":"62836109-e2e3-4959-8013-c742aa942aea","Type":"ContainerStarted","Data":"ffd19112bf4f45571dcc0127b989215fa50e7b0385556b506e7781218d8308dd"}
Jul 31 14:57:23.512063 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.511988    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:57:23.527263 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.527213    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-ngp59" podStartSLOduration=2.5981013539999998 podStartE2EDuration="5.527178039s" totalImagesPullingTime="2.929076685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:23.527135863 +0000 UTC m=+555.481449766" watchObservedRunningTime="2026-07-31 14:57:23.527178039 +0000 UTC m=+555.481491927"
Jul 31 14:57:23.545421 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:23.545371    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-7vvtz" podStartSLOduration=2.003241937 podStartE2EDuration="5.545353151s" totalImagesPullingTime="3.542111214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:23.544990777 +0000 UTC m=+555.499304680" watchObservedRunningTime="2026-07-31 14:57:23.545353151 +0000 UTC m=+555.499667058"
Jul 31 14:57:34.518713 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:34.518682    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-ngp59"
Jul 31 14:57:34.520608 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:57:34.520589    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-7vvtz"
Jul 31 14:58:08.442459 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:58:08.442432    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:58:08.451414 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:58:08.451388    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 14:59:05.630522 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.630484    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 14:59:05.633781 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.633763    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.636466 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.636447    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 14:59:05.636587 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.636469    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 14:59:05.636587 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.636446    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e9535-predictor-serving-cert\""
Jul 31 14:59:05.637415 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.637397    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e9535-kube-rbac-proxy-sar-config\""
Jul 31 14:59:05.637491 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.637399    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hpn59\""
Jul 31 14:59:05.643311 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.643292    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 14:59:05.711041 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.711007    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.711230 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.711046    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.711230 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.711079    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.811943 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.811902    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.812121 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.811970    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.812121 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.812012    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.812265 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:59:05.812149    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e9535-predictor-serving-cert: secret "success-200-isvc-e9535-predictor-serving-cert" not found
Jul 31 14:59:05.812321 ip-10-0-135-216 kubenswrapper[2568]: E0731 14:59:05.812267    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls podName:fe678310-682a-47b6-a15d-ec4e62263f98 nodeName:}" failed. No retries permitted until 2026-07-31 14:59:06.312245455 +0000 UTC m=+658.266559342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls") pod "success-200-isvc-e9535-predictor-7cdddc598d-krch6" (UID: "fe678310-682a-47b6-a15d-ec4e62263f98") : secret "success-200-isvc-e9535-predictor-serving-cert" not found
Jul 31 14:59:05.812727 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.812703    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:05.821327 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:05.821298    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:06.152345 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.152308    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 14:59:06.155791 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.155769    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.159307 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.159284    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e9535-kube-rbac-proxy-sar-config\""
Jul 31 14:59:06.159806 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.159789    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e9535-predictor-serving-cert\""
Jul 31 14:59:06.189089 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.189058    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 14:59:06.214485 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.214448    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.214636 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.214516    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsgk\" (UniqueName: \"kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.214636 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.214566    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.315143 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.315111    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.315322 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.315179    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.315322 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.315241    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:06.315322 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.315291    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsgk\" (UniqueName: \"kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.316001 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.315892    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.317769 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.317741    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") pod \"success-200-isvc-e9535-predictor-7cdddc598d-krch6\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") " pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:06.317877 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.317780    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.339001 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.338981    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsgk\" (UniqueName: \"kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk\") pod \"error-404-isvc-e9535-predictor-54f44fc89c-p24vc\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") " pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.465272 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.465240    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:06.544296 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.544265    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:06.621537 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.621513    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 14:59:06.623867 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:59:06.623841    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d60a8c_0e8f_4a59_ad42_c1cbf1819e18.slice/crio-39d34f8be88cca6427c1a488254af76443ae3f2bde1c842455b00a80a79ac979 WatchSource:0}: Error finding container 39d34f8be88cca6427c1a488254af76443ae3f2bde1c842455b00a80a79ac979: Status 404 returned error can't find the container with id 39d34f8be88cca6427c1a488254af76443ae3f2bde1c842455b00a80a79ac979
Jul 31 14:59:06.701946 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.701917    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 14:59:06.705359 ip-10-0-135-216 kubenswrapper[2568]: W0731 14:59:06.705332    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe678310_682a_47b6_a15d_ec4e62263f98.slice/crio-3e17d66502ad084dbc49c9f6993bbf7d78c6090fcec2e66f33f060d901420c68 WatchSource:0}: Error finding container 3e17d66502ad084dbc49c9f6993bbf7d78c6090fcec2e66f33f060d901420c68: Status 404 returned error can't find the container with id 3e17d66502ad084dbc49c9f6993bbf7d78c6090fcec2e66f33f060d901420c68
Jul 31 14:59:06.863961 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.863867    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerStarted","Data":"3e17d66502ad084dbc49c9f6993bbf7d78c6090fcec2e66f33f060d901420c68"}
Jul 31 14:59:06.865067 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:06.865040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerStarted","Data":"39d34f8be88cca6427c1a488254af76443ae3f2bde1c842455b00a80a79ac979"}
Jul 31 14:59:22.957537 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:22.957499    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerStarted","Data":"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"}
Jul 31 14:59:22.958887 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:22.958866    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerStarted","Data":"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"}
Jul 31 14:59:25.974621 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.974579    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerStarted","Data":"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"}
Jul 31 14:59:25.975081 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.974708    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:25.976344 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.976322    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerStarted","Data":"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"}
Jul 31 14:59:25.976532 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.976518    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:25.976574 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.976548    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:25.977860 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.977828    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 14:59:25.995242 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:25.995171    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podStartSLOduration=2.428783798 podStartE2EDuration="20.995157091s" totalImagesPullingTime="18.566373293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:25.993344616 +0000 UTC m=+677.947658525" watchObservedRunningTime="2026-07-31 14:59:25.995157091 +0000 UTC m=+677.949470997"
Jul 31 14:59:26.016119 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:26.016071    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podStartSLOduration=1.385937169 podStartE2EDuration="20.016055511s" totalImagesPullingTime="18.630118342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:26.015119936 +0000 UTC m=+677.969433844" watchObservedRunningTime="2026-07-31 14:59:26.016055511 +0000 UTC m=+677.970369417"
Jul 31 14:59:26.980434 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:26.980402    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:26.980901 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:26.980402    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 14:59:26.981576 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:26.981556    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 14:59:27.983709 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:27.983670    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 14:59:31.984587 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:31.984560    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 14:59:31.984962 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:31.984932    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 14:59:32.988079 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:32.988052    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 14:59:32.988625 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:32.988602    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 14:59:41.985262 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:41.985213    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 14:59:42.989294 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:42.989242    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 14:59:51.985543 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:51.985503    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 14:59:52.988797 ip-10-0-135-216 kubenswrapper[2568]: I0731 14:59:52.988758    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 15:00:01.985165 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:01.985066    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 15:00:02.988677 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:02.988633    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 15:00:11.985345 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:11.985316    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 15:00:12.990027 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:12.989999    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 15:00:40.212437 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.212403    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:00:40.215689 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.215672    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.218482 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.218462    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b0b5-predictor-serving-cert\""
Jul 31 15:00:40.218482 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.218468    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b0b5-kube-rbac-proxy-sar-config\""
Jul 31 15:00:40.230683 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.230660    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:00:40.250079 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.250051    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 15:00:40.250342 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.250302    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" containerID="cri-o://1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799" gracePeriod=30
Jul 31 15:00:40.250409 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.250333    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kube-rbac-proxy" containerID="cri-o://fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc" gracePeriod=30
Jul 31 15:00:40.306724 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.306690    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:00:40.309966 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.309949    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.313031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.313010    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b0b5-kube-rbac-proxy-sar-config\""
Jul 31 15:00:40.313261 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.313238    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b0b5-predictor-serving-cert\""
Jul 31 15:00:40.321953 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.321934    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:00:40.359014 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.358980    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l2zv\" (UniqueName: \"kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.359224 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.359022    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbnn\" (UniqueName: \"kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.359224 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.359060    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.359224 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.359091    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.359224 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.359157    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.359224 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.359186    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.368268 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.368085    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 15:00:40.368422 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.368396    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" containerID="cri-o://775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140" gracePeriod=30
Jul 31 15:00:40.368519 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.368494    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kube-rbac-proxy" containerID="cri-o://07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa" gracePeriod=30
Jul 31 15:00:40.460311 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460267    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.460513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460322    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.460513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460356    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2zv\" (UniqueName: \"kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.460513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460400    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbnn\" (UniqueName: \"kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.460513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460423    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.460513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.460477    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.461176 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.461144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.461324 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.461182    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.462966 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.462928    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.463033 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.463018    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.468913 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.468888    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l2zv\" (UniqueName: \"kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv\") pod \"error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") " pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.469077 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.469007    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbnn\" (UniqueName: \"kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn\") pod \"success-200-isvc-7b0b5-predictor-df99ffc46-svnw2\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") " pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.527648 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.527612    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:40.620677 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.620650    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:40.665940 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.665904    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:00:40.668709 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:00:40.668677    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9022088_6ae0_450a_b0e6_abafe9638ffb.slice/crio-1157dca9c76b4cefdc1a04a7240d058fa3c17c94f06ccaf50543bc164a4f16fb WatchSource:0}: Error finding container 1157dca9c76b4cefdc1a04a7240d058fa3c17c94f06ccaf50543bc164a4f16fb: Status 404 returned error can't find the container with id 1157dca9c76b4cefdc1a04a7240d058fa3c17c94f06ccaf50543bc164a4f16fb
Jul 31 15:00:40.758554 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:40.758532    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:00:40.761763 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:00:40.761737    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7735513_ebb7_4d09_baa6_7b74d68c942f.slice/crio-d42e75a6072c0c0d4d77c1a8f6299de2b0f45ce108158bd2ee2dfad59c521ad1 WatchSource:0}: Error finding container d42e75a6072c0c0d4d77c1a8f6299de2b0f45ce108158bd2ee2dfad59c521ad1: Status 404 returned error can't find the container with id d42e75a6072c0c0d4d77c1a8f6299de2b0f45ce108158bd2ee2dfad59c521ad1
Jul 31 15:00:41.245405 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.245360    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerStarted","Data":"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"}
Jul 31 15:00:41.245876 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.245478    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:41.245876 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.245510    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:41.245876 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.245534    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerStarted","Data":"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"}
Jul 31 15:00:41.245876 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.245566    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerStarted","Data":"d42e75a6072c0c0d4d77c1a8f6299de2b0f45ce108158bd2ee2dfad59c521ad1"}
Jul 31 15:00:41.247062 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.247030    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:41.247179 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.247161    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerStarted","Data":"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"}
Jul 31 15:00:41.247274 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.247234    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerStarted","Data":"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"}
Jul 31 15:00:41.247274 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.247270    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:41.247362 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.247298    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerStarted","Data":"1157dca9c76b4cefdc1a04a7240d058fa3c17c94f06ccaf50543bc164a4f16fb"}
Jul 31 15:00:41.248899 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.248877    2568 generic.go:356] "Generic (PLEG): container finished" podID="fe678310-682a-47b6-a15d-ec4e62263f98" containerID="fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc" exitCode=2
Jul 31 15:00:41.248993 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.248952    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerDied","Data":"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"}
Jul 31 15:00:41.250634 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.250614    2568 generic.go:356] "Generic (PLEG): container finished" podID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerID="07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa" exitCode=2
Jul 31 15:00:41.250715 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.250645    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerDied","Data":"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"}
Jul 31 15:00:41.265912 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.265870    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podStartSLOduration=1.265856796 podStartE2EDuration="1.265856796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:41.264097323 +0000 UTC m=+753.218411295" watchObservedRunningTime="2026-07-31 15:00:41.265856796 +0000 UTC m=+753.220170701"
Jul 31 15:00:41.282857 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.282804    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podStartSLOduration=1.282786424 podStartE2EDuration="1.282786424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:00:41.282745609 +0000 UTC m=+753.237059528" watchObservedRunningTime="2026-07-31 15:00:41.282786424 +0000 UTC m=+753.237100332"
Jul 31 15:00:41.981455 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.981408    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 31 15:00:41.985732 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:41.985711    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 31 15:00:42.254452 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:42.254361    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:42.254452 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:42.254385    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:42.255409 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:42.255383    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:00:42.984280 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:42.984238    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 31 15:00:42.988747 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:42.988724    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 31 15:00:43.260310 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.260182    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:00:43.595895 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.595865    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 15:00:43.687489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.687461    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") pod \"fe678310-682a-47b6-a15d-ec4e62263f98\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") "
Jul 31 15:00:43.687595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.687518    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2\") pod \"fe678310-682a-47b6-a15d-ec4e62263f98\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") "
Jul 31 15:00:43.687595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.687585    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config\") pod \"fe678310-682a-47b6-a15d-ec4e62263f98\" (UID: \"fe678310-682a-47b6-a15d-ec4e62263f98\") "
Jul 31 15:00:43.687930 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.687904    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe678310-682a-47b6-a15d-ec4e62263f98" (UID: "fe678310-682a-47b6-a15d-ec4e62263f98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:00:43.689729 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.689703    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2" (OuterVolumeSpecName: "kube-api-access-7pqk2") pod "fe678310-682a-47b6-a15d-ec4e62263f98" (UID: "fe678310-682a-47b6-a15d-ec4e62263f98"). InnerVolumeSpecName "kube-api-access-7pqk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:00:43.689817 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.689706    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe678310-682a-47b6-a15d-ec4e62263f98" (UID: "fe678310-682a-47b6-a15d-ec4e62263f98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:00:43.788853 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.788777    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe678310-682a-47b6-a15d-ec4e62263f98-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:43.788853 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.788803    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/fe678310-682a-47b6-a15d-ec4e62263f98-kube-api-access-7pqk2\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:43.788853 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.788815    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe678310-682a-47b6-a15d-ec4e62263f98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:43.804124 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.804101    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 15:00:43.889093 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.889059    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls\") pod \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") "
Jul 31 15:00:43.889279 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.889111    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config\") pod \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") "
Jul 31 15:00:43.889279 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.889131    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjsgk\" (UniqueName: \"kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk\") pod \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\" (UID: \"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18\") "
Jul 31 15:00:43.889455 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.889432    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" (UID: "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:00:43.891406 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.891378    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk" (OuterVolumeSpecName: "kube-api-access-tjsgk") pod "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" (UID: "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18"). InnerVolumeSpecName "kube-api-access-tjsgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:00:43.891506 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.891413    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" (UID: "43d60a8c-0e8f-4a59-ad42-c1cbf1819e18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:00:43.989741 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.989707    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:43.989741 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.989734    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsgk\" (UniqueName: \"kubernetes.io/projected/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-kube-api-access-tjsgk\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:43.989741 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:43.989744    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:00:44.266086 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.266046    2568 generic.go:356] "Generic (PLEG): container finished" podID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerID="775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140" exitCode=0
Jul 31 15:00:44.266498 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.266100    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"
Jul 31 15:00:44.266498 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.266103    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerDied","Data":"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"}
Jul 31 15:00:44.266498 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.266172    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc" event={"ID":"43d60a8c-0e8f-4a59-ad42-c1cbf1819e18","Type":"ContainerDied","Data":"39d34f8be88cca6427c1a488254af76443ae3f2bde1c842455b00a80a79ac979"}
Jul 31 15:00:44.266498 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.266226    2568 scope.go:117] "RemoveContainer" containerID="07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"
Jul 31 15:00:44.268314 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.268293    2568 generic.go:356] "Generic (PLEG): container finished" podID="fe678310-682a-47b6-a15d-ec4e62263f98" containerID="1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799" exitCode=0
Jul 31 15:00:44.268448 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.268343    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerDied","Data":"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"}
Jul 31 15:00:44.268448 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.268365    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"
Jul 31 15:00:44.268448 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.268389    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6" event={"ID":"fe678310-682a-47b6-a15d-ec4e62263f98","Type":"ContainerDied","Data":"3e17d66502ad084dbc49c9f6993bbf7d78c6090fcec2e66f33f060d901420c68"}
Jul 31 15:00:44.276627 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.276608    2568 scope.go:117] "RemoveContainer" containerID="775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"
Jul 31 15:00:44.285266 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.285248    2568 scope.go:117] "RemoveContainer" containerID="07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"
Jul 31 15:00:44.285528 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:00:44.285508    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa\": container with ID starting with 07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa not found: ID does not exist" containerID="07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"
Jul 31 15:00:44.285577 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.285539    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa"} err="failed to get container status \"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa\": rpc error: code = NotFound desc = could not find container \"07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa\": container with ID starting with 07552bd41a1fe18ce941e6b5aa442120df44f98e05376a2e2fc1abd04b5813fa not found: ID does not exist"
Jul 31 15:00:44.285577 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.285568    2568 scope.go:117] "RemoveContainer" containerID="775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"
Jul 31 15:00:44.285812 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:00:44.285796    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140\": container with ID starting with 775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140 not found: ID does not exist" containerID="775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"
Jul 31 15:00:44.285868 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.285813    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140"} err="failed to get container status \"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140\": rpc error: code = NotFound desc = could not find container \"775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140\": container with ID starting with 775e7914a2ed42c46e92ddd38b3ada031971e90492a83a0afde64db26cc3b140 not found: ID does not exist"
Jul 31 15:00:44.285868 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.285825    2568 scope.go:117] "RemoveContainer" containerID="fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"
Jul 31 15:00:44.291707 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.291686    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 15:00:44.294155 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.294139    2568 scope.go:117] "RemoveContainer" containerID="1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"
Jul 31 15:00:44.294524 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.294505    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e9535-predictor-54f44fc89c-p24vc"]
Jul 31 15:00:44.302543 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.302518    2568 scope.go:117] "RemoveContainer" containerID="fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"
Jul 31 15:00:44.302839 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:00:44.302820    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc\": container with ID starting with fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc not found: ID does not exist" containerID="fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"
Jul 31 15:00:44.302907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.302847    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc"} err="failed to get container status \"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc\": rpc error: code = NotFound desc = could not find container \"fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc\": container with ID starting with fea02a951b9cf379a8a25d375c40fd8614a8ccc683d4723dab916b71517fdbcc not found: ID does not exist"
Jul 31 15:00:44.302907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.302865    2568 scope.go:117] "RemoveContainer" containerID="1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"
Jul 31 15:00:44.303100 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:00:44.303078    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799\": container with ID starting with 1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799 not found: ID does not exist" containerID="1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"
Jul 31 15:00:44.303151 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.303104    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799"} err="failed to get container status \"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799\": rpc error: code = NotFound desc = could not find container \"1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799\": container with ID starting with 1ec8c2a0b10eb27e1db64451a8327e50ca7411047cf85a4c7ecb65ec69631799 not found: ID does not exist"
Jul 31 15:00:44.310572 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.310550    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 15:00:44.314480 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.314461    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e9535-predictor-7cdddc598d-krch6"]
Jul 31 15:00:44.588815 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.588740    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" path="/var/lib/kubelet/pods/43d60a8c-0e8f-4a59-ad42-c1cbf1819e18/volumes"
Jul 31 15:00:44.589144 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:44.589131    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" path="/var/lib/kubelet/pods/fe678310-682a-47b6-a15d-ec4e62263f98/volumes"
Jul 31 15:00:47.258874 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:47.258846    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:00:47.259317 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:47.259290    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:48.265274 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:48.265245    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:00:48.265779 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:48.265749    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:00:57.259321 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:57.259279    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:58.266510 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:00:58.266471    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:01:07.259458 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:07.259406    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:01:08.266333 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:08.266294    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:01:17.259812 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:17.259764    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:01:18.266631 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:18.266594    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:01:27.259867 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:27.259838    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:01:28.267126 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:28.267098    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:01:35.987621 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987535    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987841    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kube-rbac-proxy"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987851    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kube-rbac-proxy"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987873    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kube-rbac-proxy"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987880    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kube-rbac-proxy"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987893    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987898    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987906    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987911    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987963    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987971    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kube-rbac-proxy"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987977    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d60a8c-0e8f-4a59-ad42-c1cbf1819e18" containerName="kserve-container"
Jul 31 15:01:35.988078 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.987983    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe678310-682a-47b6-a15d-ec4e62263f98" containerName="kube-rbac-proxy"
Jul 31 15:01:35.990818 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.990802    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:35.993482 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.993453    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a9565-predictor-serving-cert\""
Jul 31 15:01:35.993623 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:35.993457    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a9565-kube-rbac-proxy-sar-config\""
Jul 31 15:01:36.006622 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.006592    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:01:36.075455 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.075402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.075679 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.075499    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lflhz\" (UniqueName: \"kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.075679 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.075552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.100580 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.100544    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:01:36.103829 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.103812    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.106335 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.106311    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9565-kube-rbac-proxy-sar-config\""
Jul 31 15:01:36.106450 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.106363    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a9565-predictor-serving-cert\""
Jul 31 15:01:36.116038 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.116017    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:01:36.176332 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176298    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lflhz\" (UniqueName: \"kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.176332 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176333    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdzl\" (UniqueName: \"kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.176587 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.176587 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176383    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.176587 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176425    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.176587 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.176463    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.177051 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.177033    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.178924 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.178903    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.185426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.185394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lflhz\" (UniqueName: \"kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz\") pod \"success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") " pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.277499 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.277410    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdzl\" (UniqueName: \"kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.277499 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.277463    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.277765 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.277523    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.278399 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.278365    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.280488 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.280466    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.287162 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.287144    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdzl\" (UniqueName: \"kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl\") pod \"error-404-isvc-a9565-predictor-7689f7fc67-cvwzg\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") " pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.300971 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.300945    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:36.413688 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.413657    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:36.434762 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.434733    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:01:36.437982 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:01:36.437941    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0512425d_0b38_4e37_9c0c_0fe6fbf49e17.slice/crio-918b56780740a0158ad5467b83a1e03cc69e9b89f9a4947344f719f09aa74f29 WatchSource:0}: Error finding container 918b56780740a0158ad5467b83a1e03cc69e9b89f9a4947344f719f09aa74f29: Status 404 returned error can't find the container with id 918b56780740a0158ad5467b83a1e03cc69e9b89f9a4947344f719f09aa74f29
Jul 31 15:01:36.466324 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.466017    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerStarted","Data":"918b56780740a0158ad5467b83a1e03cc69e9b89f9a4947344f719f09aa74f29"}
Jul 31 15:01:36.553413 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:36.553388    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:01:36.555573 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:01:36.555547    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb998829_e9e8_4c7e_a2c1_b0ac62c98c74.slice/crio-8efd4a31f6191d7970f3e0b0b1130b772e57e302542e5a93895910da46c52081 WatchSource:0}: Error finding container 8efd4a31f6191d7970f3e0b0b1130b772e57e302542e5a93895910da46c52081: Status 404 returned error can't find the container with id 8efd4a31f6191d7970f3e0b0b1130b772e57e302542e5a93895910da46c52081
Jul 31 15:01:37.471897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.471861    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerStarted","Data":"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"}
Jul 31 15:01:37.472422 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.471924    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerStarted","Data":"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"}
Jul 31 15:01:37.472422 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.471957    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:37.472422 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.471988    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:37.473734 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.473698    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:01:37.473878 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.473778    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerStarted","Data":"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"}
Jul 31 15:01:37.473878 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.473831    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerStarted","Data":"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"}
Jul 31 15:01:37.473878 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.473862    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerStarted","Data":"8efd4a31f6191d7970f3e0b0b1130b772e57e302542e5a93895910da46c52081"}
Jul 31 15:01:37.474011 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.473975    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:37.494445 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.494406    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podStartSLOduration=2.494393188 podStartE2EDuration="2.494393188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:37.492428796 +0000 UTC m=+809.446742700" watchObservedRunningTime="2026-07-31 15:01:37.494393188 +0000 UTC m=+809.448707148"
Jul 31 15:01:37.513959 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:37.513916    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podStartSLOduration=1.5139009209999998 podStartE2EDuration="1.513900921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:37.513349322 +0000 UTC m=+809.467663228" watchObservedRunningTime="2026-07-31 15:01:37.513900921 +0000 UTC m=+809.468214828"
Jul 31 15:01:38.478022 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:38.477985    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:38.478437 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:38.478022    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:01:38.479315 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:38.479291    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:39.481692 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:39.481652    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:43.482446 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:43.482417    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:01:43.482949 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:43.482823    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:01:44.486854 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:44.486823    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:01:44.487414 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:44.487384    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:53.483287 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:53.483242    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:01:54.487433 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:01:54.487395    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:02:03.483818 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:03.483778    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:02:04.487346 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:04.487308    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:02:13.482852 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:13.482808    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:02:14.487876 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:14.487838    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:02:23.483348 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:23.483320    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:02:24.487889 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:02:24.487854    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:03:08.470324 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:03:08.470242    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:03:08.473066 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:03:08.473042    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:08:08.499479 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:08:08.499452    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:08:08.500058 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:08:08.499457    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:09:54.945470 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.945425    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:09:54.948055 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.945662    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" containerID="cri-o://3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48" gracePeriod=30
Jul 31 15:09:54.948055 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.945770    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kube-rbac-proxy" containerID="cri-o://1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b" gracePeriod=30
Jul 31 15:09:54.988584 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.988552    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:09:54.991865 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.991850    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:54.994816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.994784    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49957-predictor-serving-cert\""
Jul 31 15:09:54.994978 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:54.994898    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-49957-kube-rbac-proxy-sar-config\""
Jul 31 15:09:55.002210 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.002155    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:09:55.058449 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.058417    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:09:55.061867 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.061848    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.064560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.064539    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49957-predictor-serving-cert\""
Jul 31 15:09:55.064560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.064553    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49957-kube-rbac-proxy-sar-config\""
Jul 31 15:09:55.072621 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.072597    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:09:55.106477 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106444    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.106663 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106489    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfg66\" (UniqueName: \"kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.106663 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.106663 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wwh\" (UniqueName: \"kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.106830 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106690    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.106830 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.106743    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.116226 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.116182    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:09:55.116489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.116460    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kube-rbac-proxy" containerID="cri-o://4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5" gracePeriod=30
Jul 31 15:09:55.116489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.116457    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" containerID="cri-o://cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2" gracePeriod=30
Jul 31 15:09:55.208134 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208029    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.208134 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208079    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfg66\" (UniqueName: \"kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.208134 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.208134 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208132    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wwh\" (UniqueName: \"kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.208508 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208254    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.208508 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208286    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.208825 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208798    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.208952 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.208854    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.210815 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.210788    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.210907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.210819    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.215649 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.215619    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfg66\" (UniqueName: \"kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66\") pod \"success-200-isvc-49957-predictor-c6d6b9746-75srq\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") " pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.215763 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.215743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wwh\" (UniqueName: \"kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh\") pod \"error-404-isvc-49957-predictor-bc66879c5-mgq5g\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") " pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.303054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.303015    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:55.314677 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.314644    2568 generic.go:356] "Generic (PLEG): container finished" podID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerID="4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5" exitCode=2
Jul 31 15:09:55.314802 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.314712    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerDied","Data":"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"}
Jul 31 15:09:55.316702 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.316679    2568 generic.go:356] "Generic (PLEG): container finished" podID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerID="1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b" exitCode=2
Jul 31 15:09:55.316811 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.316746    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerDied","Data":"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"}
Jul 31 15:09:55.372398 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.372362    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:55.447854 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.447825    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:09:55.451037 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:09:55.451004    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b69d458_b631_4a3c_a66f_f7f5b63d59b3.slice/crio-d495ede97cac079023e0465d7a393dfa326715271bc80dd9b5e91759143a8bcb WatchSource:0}: Error finding container d495ede97cac079023e0465d7a393dfa326715271bc80dd9b5e91759143a8bcb: Status 404 returned error can't find the container with id d495ede97cac079023e0465d7a393dfa326715271bc80dd9b5e91759143a8bcb
Jul 31 15:09:55.515409 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:55.515382    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:09:55.518171 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:09:55.518140    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251c1551_b708_4f55_a230_5578c7409efc.slice/crio-e7f3b6e1c3e3dfcca67e958b975697ad09a5a1968d6eed36c441e24942ca1a63 WatchSource:0}: Error finding container e7f3b6e1c3e3dfcca67e958b975697ad09a5a1968d6eed36c441e24942ca1a63: Status 404 returned error can't find the container with id e7f3b6e1c3e3dfcca67e958b975697ad09a5a1968d6eed36c441e24942ca1a63
Jul 31 15:09:56.322073 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.322033    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerStarted","Data":"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"}
Jul 31 15:09:56.322552 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.322144    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:56.322552 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.322178    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:09:56.322552 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.322222    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerStarted","Data":"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"}
Jul 31 15:09:56.322552 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.322244    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerStarted","Data":"d495ede97cac079023e0465d7a393dfa326715271bc80dd9b5e91759143a8bcb"}
Jul 31 15:09:56.323544 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323514    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:09:56.323893 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323864    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerStarted","Data":"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"}
Jul 31 15:09:56.323969 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323924    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:56.323969 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323946    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerStarted","Data":"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"}
Jul 31 15:09:56.324065 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323974    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerStarted","Data":"e7f3b6e1c3e3dfcca67e958b975697ad09a5a1968d6eed36c441e24942ca1a63"}
Jul 31 15:09:56.324065 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.323995    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:09:56.325098 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.325076    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:09:56.341754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.341710    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podStartSLOduration=2.341695373 podStartE2EDuration="2.341695373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:56.339807389 +0000 UTC m=+1308.294121295" watchObservedRunningTime="2026-07-31 15:09:56.341695373 +0000 UTC m=+1308.296009278"
Jul 31 15:09:56.360064 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:56.360005    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podStartSLOduration=1.359989181 podStartE2EDuration="1.359989181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:56.359602542 +0000 UTC m=+1308.313916449" watchObservedRunningTime="2026-07-31 15:09:56.359989181 +0000 UTC m=+1308.314303091"
Jul 31 15:09:57.255470 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:57.255422    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:09:57.259813 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:57.259784    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:09:57.327622 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:57.327580    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:09:57.328011 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:57.327678    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:09:58.261310 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:58.261262    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 31 15:09:58.265761 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:58.265734    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 31 15:09:58.993103 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:58.993078    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:09:59.034873 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.034845    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls\") pod \"b9022088-6ae0-450a-b0e6-abafe9638ffb\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") "
Jul 31 15:09:59.035008 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.034902    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config\") pod \"b9022088-6ae0-450a-b0e6-abafe9638ffb\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") "
Jul 31 15:09:59.035008 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.034939    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cbnn\" (UniqueName: \"kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn\") pod \"b9022088-6ae0-450a-b0e6-abafe9638ffb\" (UID: \"b9022088-6ae0-450a-b0e6-abafe9638ffb\") "
Jul 31 15:09:59.035359 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.035329    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9022088-6ae0-450a-b0e6-abafe9638ffb" (UID: "b9022088-6ae0-450a-b0e6-abafe9638ffb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:09:59.037020 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.036996    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn" (OuterVolumeSpecName: "kube-api-access-5cbnn") pod "b9022088-6ae0-450a-b0e6-abafe9638ffb" (UID: "b9022088-6ae0-450a-b0e6-abafe9638ffb"). InnerVolumeSpecName "kube-api-access-5cbnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:09:59.037079 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.037062    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9022088-6ae0-450a-b0e6-abafe9638ffb" (UID: "b9022088-6ae0-450a-b0e6-abafe9638ffb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:09:59.135890 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.135861    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9022088-6ae0-450a-b0e6-abafe9638ffb-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.135890 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.135888    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.135890 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.135900    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbnn\" (UniqueName: \"kubernetes.io/projected/b9022088-6ae0-450a-b0e6-abafe9638ffb-kube-api-access-5cbnn\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.148045 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.148020    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:09:59.236132 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.236098    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config\") pod \"f7735513-ebb7-4d09-baa6-7b74d68c942f\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") "
Jul 31 15:09:59.236330 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.236153    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l2zv\" (UniqueName: \"kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv\") pod \"f7735513-ebb7-4d09-baa6-7b74d68c942f\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") "
Jul 31 15:09:59.236330 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.236264    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls\") pod \"f7735513-ebb7-4d09-baa6-7b74d68c942f\" (UID: \"f7735513-ebb7-4d09-baa6-7b74d68c942f\") "
Jul 31 15:09:59.236548 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.236530    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7735513-ebb7-4d09-baa6-7b74d68c942f" (UID: "f7735513-ebb7-4d09-baa6-7b74d68c942f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:09:59.238373 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.238347    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv" (OuterVolumeSpecName: "kube-api-access-6l2zv") pod "f7735513-ebb7-4d09-baa6-7b74d68c942f" (UID: "f7735513-ebb7-4d09-baa6-7b74d68c942f"). InnerVolumeSpecName "kube-api-access-6l2zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:09:59.238437 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.238387    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7735513-ebb7-4d09-baa6-7b74d68c942f" (UID: "f7735513-ebb7-4d09-baa6-7b74d68c942f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:09:59.336133 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336038    2568 generic.go:356] "Generic (PLEG): container finished" podID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerID="cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2" exitCode=0
Jul 31 15:09:59.336133 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336096    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"
Jul 31 15:09:59.336359 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336131    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerDied","Data":"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"}
Jul 31 15:09:59.336359 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336238    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx" event={"ID":"f7735513-ebb7-4d09-baa6-7b74d68c942f","Type":"ContainerDied","Data":"d42e75a6072c0c0d4d77c1a8f6299de2b0f45ce108158bd2ee2dfad59c521ad1"}
Jul 31 15:09:59.336359 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336281    2568 scope.go:117] "RemoveContainer" containerID="4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"
Jul 31 15:09:59.336845 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336820    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7735513-ebb7-4d09-baa6-7b74d68c942f-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.336845 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336849    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.337038 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.336865    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l2zv\" (UniqueName: \"kubernetes.io/projected/f7735513-ebb7-4d09-baa6-7b74d68c942f-kube-api-access-6l2zv\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:09:59.337828 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.337802    2568 generic.go:356] "Generic (PLEG): container finished" podID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerID="3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48" exitCode=0
Jul 31 15:09:59.337923 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.337848    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"
Jul 31 15:09:59.338012 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.337901    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerDied","Data":"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"}
Jul 31 15:09:59.338072 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.338059    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2" event={"ID":"b9022088-6ae0-450a-b0e6-abafe9638ffb","Type":"ContainerDied","Data":"1157dca9c76b4cefdc1a04a7240d058fa3c17c94f06ccaf50543bc164a4f16fb"}
Jul 31 15:09:59.345378 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.345319    2568 scope.go:117] "RemoveContainer" containerID="cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"
Jul 31 15:09:59.352741 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.352725    2568 scope.go:117] "RemoveContainer" containerID="4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"
Jul 31 15:09:59.353014 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:09:59.352995    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5\": container with ID starting with 4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5 not found: ID does not exist" containerID="4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"
Jul 31 15:09:59.353069 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.353022    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5"} err="failed to get container status \"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5\": rpc error: code = NotFound desc = could not find container \"4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5\": container with ID starting with 4c365a36656864f7b3b676af8fdd13cf695db779432bf0513377d1b24cfce0d5 not found: ID does not exist"
Jul 31 15:09:59.353069 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.353038    2568 scope.go:117] "RemoveContainer" containerID="cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"
Jul 31 15:09:59.353299 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:09:59.353283    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2\": container with ID starting with cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2 not found: ID does not exist" containerID="cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"
Jul 31 15:09:59.353346 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.353306    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2"} err="failed to get container status \"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2\": rpc error: code = NotFound desc = could not find container \"cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2\": container with ID starting with cfcf1f41b20290e0935fb8c6711e716719fc33b9a2ad362848719c56f6baa3b2 not found: ID does not exist"
Jul 31 15:09:59.353346 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.353321    2568 scope.go:117] "RemoveContainer" containerID="1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"
Jul 31 15:09:59.360928 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.360909    2568 scope.go:117] "RemoveContainer" containerID="3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"
Jul 31 15:09:59.361747 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.361724    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:09:59.365895 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.365873    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b0b5-predictor-fc66d6ff-kgclx"]
Jul 31 15:09:59.368920 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.368902    2568 scope.go:117] "RemoveContainer" containerID="1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"
Jul 31 15:09:59.369168 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:09:59.369148    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b\": container with ID starting with 1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b not found: ID does not exist" containerID="1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"
Jul 31 15:09:59.369273 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.369175    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b"} err="failed to get container status \"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b\": rpc error: code = NotFound desc = could not find container \"1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b\": container with ID starting with 1c4ed996ded2ea14f5c3676231f689bd81270f979f6112862d61b4609e19a27b not found: ID does not exist"
Jul 31 15:09:59.369320 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.369276    2568 scope.go:117] "RemoveContainer" containerID="3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"
Jul 31 15:09:59.369508 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:09:59.369491    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48\": container with ID starting with 3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48 not found: ID does not exist" containerID="3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"
Jul 31 15:09:59.369554 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.369517    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48"} err="failed to get container status \"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48\": rpc error: code = NotFound desc = could not find container \"3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48\": container with ID starting with 3e56242f919e5ed11acae1cd13b189720e63b40c787d31adf85ba41b35e1dc48 not found: ID does not exist"
Jul 31 15:09:59.376544 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.376522    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:09:59.389296 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:09:59.389267    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b0b5-predictor-df99ffc46-svnw2"]
Jul 31 15:10:00.589503 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:00.589467    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" path="/var/lib/kubelet/pods/b9022088-6ae0-450a-b0e6-abafe9638ffb/volumes"
Jul 31 15:10:00.589875 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:00.589860    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" path="/var/lib/kubelet/pods/f7735513-ebb7-4d09-baa6-7b74d68c942f/volumes"
Jul 31 15:10:02.331978 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:02.331950    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:10:02.332419 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:02.332387    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:10:02.332488 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:02.332416    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:10:02.332765 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:02.332740    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:10:12.332438 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:12.332393    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:10:12.332844 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:12.332818    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:10:22.333074 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:22.333036    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:10:22.333495 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:22.333034    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:10:32.333091 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:32.332996    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 31 15:10:32.333600 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:32.332996    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 31 15:10:42.333572 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:42.333543    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:10:42.334012 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:42.333635    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:10:50.846155 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.846124    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:10:50.846607 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.846381    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" containerID="cri-o://e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe" gracePeriod=30
Jul 31 15:10:50.846607 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.846461    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kube-rbac-proxy" containerID="cri-o://0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661" gracePeriod=30
Jul 31 15:10:50.912952 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.912921    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:10:50.913237 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.913153    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container" containerID="cri-o://dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee" gracePeriod=30
Jul 31 15:10:50.913381 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.913206    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kube-rbac-proxy" containerID="cri-o://0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad" gracePeriod=30
Jul 31 15:10:50.935264 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935239    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:10:50.935570 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935559    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container"
Jul 31 15:10:50.935612 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935572    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container"
Jul 31 15:10:50.935612 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935584    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kube-rbac-proxy"
Jul 31 15:10:50.935612 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935589    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kube-rbac-proxy"
Jul 31 15:10:50.935612 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935599    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kube-rbac-proxy"
Jul 31 15:10:50.935612 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935606    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kube-rbac-proxy"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935614    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935620    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935671    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kserve-container"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935679    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kserve-container"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935684    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9022088-6ae0-450a-b0e6-abafe9638ffb" containerName="kube-rbac-proxy"
Jul 31 15:10:50.935754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.935692    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7735513-ebb7-4d09-baa6-7b74d68c942f" containerName="kube-rbac-proxy"
Jul 31 15:10:50.938430 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.938415    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:50.940669 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.940646    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1442-predictor-serving-cert\""
Jul 31 15:10:50.940766 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.940672    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a1442-kube-rbac-proxy-sar-config\""
Jul 31 15:10:50.946999 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.946930    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:10:50.956950 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.956927    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64km\" (UniqueName: \"kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:50.957072 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.957008    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:50.957072 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:50.957039    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.046501 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.046470    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:10:51.049603 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.049584    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.052112 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.052093    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a1442-predictor-serving-cert\""
Jul 31 15:10:51.052267 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.052096    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a1442-kube-rbac-proxy-sar-config\""
Jul 31 15:10:51.057973 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.057942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64km\" (UniqueName: \"kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.058094 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.058056    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.058094 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.058087    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.058773 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:51.058740    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a1442-predictor-serving-cert: secret "success-200-isvc-a1442-predictor-serving-cert" not found
Jul 31 15:10:51.058881 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:51.058806    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls podName:69745f64-c3d6-4e6f-9780-54ca490f1ee2 nodeName:}" failed. No retries permitted until 2026-07-31 15:10:51.558788884 +0000 UTC m=+1363.513102784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls") pod "success-200-isvc-a1442-predictor-5759c7c998-p46m5" (UID: "69745f64-c3d6-4e6f-9780-54ca490f1ee2") : secret "success-200-isvc-a1442-predictor-serving-cert" not found
Jul 31 15:10:51.058881 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.058828    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.059976 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.059956    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:10:51.069938 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.069910    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64km\" (UniqueName: \"kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.159511 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.159410    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.159511 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.159476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrrr\" (UniqueName: \"kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.159692 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.159548    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.259870 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.259836    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrrr\" (UniqueName: \"kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.259870 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.259875    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.260113 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.259934    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.260587 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.260567    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.262425 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.262405    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.268751 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.268729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrrr\" (UniqueName: \"kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr\") pod \"error-404-isvc-a1442-predictor-84bf5c887f-bmcrh\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") " pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.360653 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.360610    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:51.494395 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.494365    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:10:51.497340 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:10:51.497297    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ffb3d0_d0f8_4504_b0a0_b5d16b80a0c0.slice/crio-cc73f6af915a5d3c4ea8db34b8c32b778b369ab5687f7ef41a4b84fe7a0347f3 WatchSource:0}: Error finding container cc73f6af915a5d3c4ea8db34b8c32b778b369ab5687f7ef41a4b84fe7a0347f3: Status 404 returned error can't find the container with id cc73f6af915a5d3c4ea8db34b8c32b778b369ab5687f7ef41a4b84fe7a0347f3
Jul 31 15:10:51.544805 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.544773    2568 generic.go:356] "Generic (PLEG): container finished" podID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerID="0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad" exitCode=2
Jul 31 15:10:51.544911 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.544835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerDied","Data":"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"}
Jul 31 15:10:51.546350 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.546322    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerStarted","Data":"cc73f6af915a5d3c4ea8db34b8c32b778b369ab5687f7ef41a4b84fe7a0347f3"}
Jul 31 15:10:51.548173 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.548153    2568 generic.go:356] "Generic (PLEG): container finished" podID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerID="0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661" exitCode=2
Jul 31 15:10:51.548295 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.548220    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerDied","Data":"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"}
Jul 31 15:10:51.562119 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.562102    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.564375 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.564351    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") pod \"success-200-isvc-a1442-predictor-5759c7c998-p46m5\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") " pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.849252 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.849184    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:51.990207 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:51.990171    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:10:51.992580 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:10:51.992540    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69745f64_c3d6_4e6f_9780_54ca490f1ee2.slice/crio-1c5f98e8e04eb76c91f5119f765c80cf36d2e8028bbabe68bcc7d985dd1f7aeb WatchSource:0}: Error finding container 1c5f98e8e04eb76c91f5119f765c80cf36d2e8028bbabe68bcc7d985dd1f7aeb: Status 404 returned error can't find the container with id 1c5f98e8e04eb76c91f5119f765c80cf36d2e8028bbabe68bcc7d985dd1f7aeb
Jul 31 15:10:52.558559 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.558520    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerStarted","Data":"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"}
Jul 31 15:10:52.558759 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.558617    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerStarted","Data":"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"}
Jul 31 15:10:52.558759 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.558726    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:52.558893 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.558801    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerStarted","Data":"1c5f98e8e04eb76c91f5119f765c80cf36d2e8028bbabe68bcc7d985dd1f7aeb"}
Jul 31 15:10:52.558893 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.558850    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:52.560129 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.560101    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:10:52.560513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.560490    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerStarted","Data":"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"}
Jul 31 15:10:52.560567 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.560544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerStarted","Data":"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"}
Jul 31 15:10:52.560611 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.560591    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:52.560650 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.560628    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:52.561679 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.561657    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:10:52.580614 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.580573    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podStartSLOduration=2.580559096 podStartE2EDuration="2.580559096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:52.578417523 +0000 UTC m=+1364.532731429" watchObservedRunningTime="2026-07-31 15:10:52.580559096 +0000 UTC m=+1364.534873002"
Jul 31 15:10:52.598999 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:52.598945    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podStartSLOduration=1.5989248109999998 podStartE2EDuration="1.598924811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:52.597302466 +0000 UTC m=+1364.551616372" watchObservedRunningTime="2026-07-31 15:10:52.598924811 +0000 UTC m=+1364.553238728"
Jul 31 15:10:53.478118 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:53.478072    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:10:53.483446 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:53.483423    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 31 15:10:53.565629 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:53.565588    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:10:53.565826 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:53.565645    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:10:54.269354 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.269328    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:10:54.283046 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.283024    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkdzl\" (UniqueName: \"kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl\") pod \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") "
Jul 31 15:10:54.283175 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.283062    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config\") pod \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") "
Jul 31 15:10:54.283175 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.283085    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls\") pod \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\" (UID: \"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74\") "
Jul 31 15:10:54.283391 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.283368    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" (UID: "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:10:54.285210 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.285175    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl" (OuterVolumeSpecName: "kube-api-access-xkdzl") pod "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" (UID: "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74"). InnerVolumeSpecName "kube-api-access-xkdzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:54.285269 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.285215    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" (UID: "cb998829-e9e8-4c7e-a2c1-b0ac62c98c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:54.384610 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.384527    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkdzl\" (UniqueName: \"kubernetes.io/projected/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-api-access-xkdzl\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:54.384610 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.384558    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:54.384610 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.384569    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:54.570128 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.570091    2568 generic.go:356] "Generic (PLEG): container finished" podID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerID="dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee" exitCode=0
Jul 31 15:10:54.570590 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.570138    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"
Jul 31 15:10:54.570590 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.570182    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerDied","Data":"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"}
Jul 31 15:10:54.570590 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.570260    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg" event={"ID":"cb998829-e9e8-4c7e-a2c1-b0ac62c98c74","Type":"ContainerDied","Data":"8efd4a31f6191d7970f3e0b0b1130b772e57e302542e5a93895910da46c52081"}
Jul 31 15:10:54.570590 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.570288    2568 scope.go:117] "RemoveContainer" containerID="0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"
Jul 31 15:10:54.580004 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.579989    2568 scope.go:117] "RemoveContainer" containerID="dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"
Jul 31 15:10:54.589000 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.588916    2568 scope.go:117] "RemoveContainer" containerID="0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"
Jul 31 15:10:54.589353 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:54.589308    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad\": container with ID starting with 0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad not found: ID does not exist" containerID="0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"
Jul 31 15:10:54.589489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.589358    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad"} err="failed to get container status \"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad\": rpc error: code = NotFound desc = could not find container \"0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad\": container with ID starting with 0725d6d07a84bdae4998f52744ea58c3c959953cfd0881d70b9fd8ba512a69ad not found: ID does not exist"
Jul 31 15:10:54.589489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.589385    2568 scope.go:117] "RemoveContainer" containerID="dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"
Jul 31 15:10:54.589873 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:54.589818    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee\": container with ID starting with dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee not found: ID does not exist" containerID="dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"
Jul 31 15:10:54.589873 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.589857    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee"} err="failed to get container status \"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee\": rpc error: code = NotFound desc = could not find container \"dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee\": container with ID starting with dcb7e3d0f637575bd05711fd73fc09f0a8f71cc0fa4eb59c8cae28b3be3187ee not found: ID does not exist"
Jul 31 15:10:54.592112 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.592093    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:10:54.595658 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.595636    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a9565-predictor-7689f7fc67-cvwzg"]
Jul 31 15:10:54.786801 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.786777    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:10:54.889883 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.889847    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lflhz\" (UniqueName: \"kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz\") pod \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") "
Jul 31 15:10:54.890085 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.889906    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls\") pod \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") "
Jul 31 15:10:54.890085 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.889933    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config\") pod \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\" (UID: \"0512425d-0b38-4e37-9c0c-0fe6fbf49e17\") "
Jul 31 15:10:54.890341 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.890316    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0512425d-0b38-4e37-9c0c-0fe6fbf49e17" (UID: "0512425d-0b38-4e37-9c0c-0fe6fbf49e17"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:10:54.892102 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.892080    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0512425d-0b38-4e37-9c0c-0fe6fbf49e17" (UID: "0512425d-0b38-4e37-9c0c-0fe6fbf49e17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:54.892165 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.892097    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz" (OuterVolumeSpecName: "kube-api-access-lflhz") pod "0512425d-0b38-4e37-9c0c-0fe6fbf49e17" (UID: "0512425d-0b38-4e37-9c0c-0fe6fbf49e17"). InnerVolumeSpecName "kube-api-access-lflhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:54.990770 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.990725    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:54.990770 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.990757    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lflhz\" (UniqueName: \"kubernetes.io/projected/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-kube-api-access-lflhz\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:54.990770 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:54.990767    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0512425d-0b38-4e37-9c0c-0fe6fbf49e17-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:10:55.575795 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.575760    2568 generic.go:356] "Generic (PLEG): container finished" podID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerID="e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe" exitCode=0
Jul 31 15:10:55.576261 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.575810    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"
Jul 31 15:10:55.576261 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.575846    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerDied","Data":"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"}
Jul 31 15:10:55.576261 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.575919    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6" event={"ID":"0512425d-0b38-4e37-9c0c-0fe6fbf49e17","Type":"ContainerDied","Data":"918b56780740a0158ad5467b83a1e03cc69e9b89f9a4947344f719f09aa74f29"}
Jul 31 15:10:55.576261 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.575948    2568 scope.go:117] "RemoveContainer" containerID="0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"
Jul 31 15:10:55.584533 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.584517    2568 scope.go:117] "RemoveContainer" containerID="e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"
Jul 31 15:10:55.591781 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.591754    2568 scope.go:117] "RemoveContainer" containerID="0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"
Jul 31 15:10:55.592038 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:55.592021    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661\": container with ID starting with 0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661 not found: ID does not exist" containerID="0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"
Jul 31 15:10:55.592095 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.592044    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661"} err="failed to get container status \"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661\": rpc error: code = NotFound desc = could not find container \"0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661\": container with ID starting with 0b192ac316a6249c538a227b773dac26e1f42898fad6fe99bcc92689d24bb661 not found: ID does not exist"
Jul 31 15:10:55.592095 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.592061    2568 scope.go:117] "RemoveContainer" containerID="e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"
Jul 31 15:10:55.592274 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:10:55.592259    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe\": container with ID starting with e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe not found: ID does not exist" containerID="e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"
Jul 31 15:10:55.592327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.592278    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe"} err="failed to get container status \"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe\": rpc error: code = NotFound desc = could not find container \"e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe\": container with ID starting with e853549edb60183c5aca9626126e229793c1919c4f019e4a1b2070e1d3f6edfe not found: ID does not exist"
Jul 31 15:10:55.598028 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.598000    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:10:55.601796 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:55.601772    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a9565-predictor-5dbbcd748b-c7sp6"]
Jul 31 15:10:56.589130 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:56.589095    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" path="/var/lib/kubelet/pods/0512425d-0b38-4e37-9c0c-0fe6fbf49e17/volumes"
Jul 31 15:10:56.589557 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:56.589543    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" path="/var/lib/kubelet/pods/cb998829-e9e8-4c7e-a2c1-b0ac62c98c74/volumes"
Jul 31 15:10:58.569901 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:58.569874    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:10:58.570322 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:58.570180    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:10:58.570373 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:58.570304    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:10:58.570633 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:10:58.570608    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:11:05.246096 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.246059    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:11:05.246654 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.246326    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container" containerID="cri-o://727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1" gracePeriod=30
Jul 31 15:11:05.246654 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.246364    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kube-rbac-proxy" containerID="cri-o://e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986" gracePeriod=30
Jul 31 15:11:05.338929 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.338891    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:11:05.339236 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339222    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339236 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339236    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339250    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339256    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339265    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339272    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339284    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339289    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339338    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339348    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kserve-container"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339356    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0512425d-0b38-4e37-9c0c-0fe6fbf49e17" containerName="kube-rbac-proxy"
Jul 31 15:11:05.339371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.339365    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb998829-e9e8-4c7e-a2c1-b0ac62c98c74" containerName="kserve-container"
Jul 31 15:11:05.343693 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.343670    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:11:05.343847 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.343823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.343934 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.343911    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container" containerID="cri-o://57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41" gracePeriod=30
Jul 31 15:11:05.344189 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.343980    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kube-rbac-proxy" containerID="cri-o://1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67" gracePeriod=30
Jul 31 15:11:05.346335 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.346312    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-06a6c-predictor-serving-cert\""
Jul 31 15:11:05.346439 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.346319    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-06a6c-kube-rbac-proxy-sar-config\""
Jul 31 15:11:05.352682 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.352660    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:11:05.373990 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.373959    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.374128 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.374043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2s2\" (UniqueName: \"kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.374128 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.374082    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.443364 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.443331    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:11:05.446918 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.446899    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.449768 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.449747    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-06a6c-predictor-serving-cert\""
Jul 31 15:11:05.453740 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.453681    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-06a6c-kube-rbac-proxy-sar-config\""
Jul 31 15:11:05.461241 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.461215    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:11:05.475796 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475773    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.475892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475803    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkph\" (UniqueName: \"kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.475892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475832    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.476001 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475913    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2s2\" (UniqueName: \"kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.476001 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.476001 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.475979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.476140 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:05.476060    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-serving-cert: secret "success-200-isvc-06a6c-predictor-serving-cert" not found
Jul 31 15:11:05.476232 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:05.476145    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls podName:5350a19f-b95b-4b88-afbb-ffa9574c5eae nodeName:}" failed. No retries permitted until 2026-07-31 15:11:05.97612645 +0000 UTC m=+1377.930440333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls") pod "success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" (UID: "5350a19f-b95b-4b88-afbb-ffa9574c5eae") : secret "success-200-isvc-06a6c-predictor-serving-cert" not found
Jul 31 15:11:05.476401 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.476384    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.488935 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.488907    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2s2\" (UniqueName: \"kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.576513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.576425    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.576513 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.576508    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.576710 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.576527    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkph\" (UniqueName: \"kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.576710 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:05.576577    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-serving-cert: secret "error-404-isvc-06a6c-predictor-serving-cert" not found
Jul 31 15:11:05.576710 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:05.576642    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls podName:e3a6010f-eee8-4a8c-894e-d5b21bec7a31 nodeName:}" failed. No retries permitted until 2026-07-31 15:11:06.0766246 +0000 UTC m=+1378.030938484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls") pod "error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" (UID: "e3a6010f-eee8-4a8c-894e-d5b21bec7a31") : secret "error-404-isvc-06a6c-predictor-serving-cert" not found
Jul 31 15:11:05.577239 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.577222    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.587212 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.587177    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkph\" (UniqueName: \"kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:05.624550 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.624515    2568 generic.go:356] "Generic (PLEG): container finished" podID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerID="e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986" exitCode=2
Jul 31 15:11:05.624714 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.624585    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerDied","Data":"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"}
Jul 31 15:11:05.626231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.626208    2568 generic.go:356] "Generic (PLEG): container finished" podID="251c1551-b708-4f55-a230-5578c7409efc" containerID="1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67" exitCode=2
Jul 31 15:11:05.626350 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.626250    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerDied","Data":"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"}
Jul 31 15:11:05.980843 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.980808    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:05.983604 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:05.983578    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") pod \"success-200-isvc-06a6c-predictor-69d6f4dc94-zf296\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") " pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:06.082170 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.082115    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:06.084692 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.084672    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") pod \"error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") " pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:06.262242 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.262142    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:06.357807 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.357775    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:06.430445 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.430418    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:11:06.432436 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:11:06.432396    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5350a19f_b95b_4b88_afbb_ffa9574c5eae.slice/crio-767f139507ae19ee2f3bc38676734e094aa0e5bf53e54d0eb7dff9b79e70c26e WatchSource:0}: Error finding container 767f139507ae19ee2f3bc38676734e094aa0e5bf53e54d0eb7dff9b79e70c26e: Status 404 returned error can't find the container with id 767f139507ae19ee2f3bc38676734e094aa0e5bf53e54d0eb7dff9b79e70c26e
Jul 31 15:11:06.507228 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.505970    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:11:06.632621 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.632584    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerStarted","Data":"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"}
Jul 31 15:11:06.632771 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.632661    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerStarted","Data":"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"}
Jul 31 15:11:06.632850 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.632786    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:06.632917 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.632863    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:06.632917 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.632895    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerStarted","Data":"767f139507ae19ee2f3bc38676734e094aa0e5bf53e54d0eb7dff9b79e70c26e"}
Jul 31 15:11:06.634021 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.633991    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:06.634497 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.634475    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerStarted","Data":"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"}
Jul 31 15:11:06.634633 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.634523    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerStarted","Data":"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"}
Jul 31 15:11:06.634633 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.634544    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerStarted","Data":"f36dcb245c03120b7d47b7b1e4989fc6b5258f0b732d4f62b00d0afe6a3cb64f"}
Jul 31 15:11:06.634633 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.634566    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:06.634633 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.634603    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:06.635729 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.635707    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:06.657172 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.657126    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podStartSLOduration=1.657110604 podStartE2EDuration="1.657110604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:06.654664419 +0000 UTC m=+1378.608978326" watchObservedRunningTime="2026-07-31 15:11:06.657110604 +0000 UTC m=+1378.611424509"
Jul 31 15:11:06.676365 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:06.676322    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podStartSLOduration=1.6763058119999998 podStartE2EDuration="1.676305812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:06.675505558 +0000 UTC m=+1378.629819463" watchObservedRunningTime="2026-07-31 15:11:06.676305812 +0000 UTC m=+1378.630619758"
Jul 31 15:11:07.328822 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:07.328769    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 31 15:11:07.329244 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:07.328769    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 31 15:11:07.638355 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:07.638250    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:07.638355 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:07.638290    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:08.570775 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:08.570746    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:11:08.571090 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:08.570743    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:11:09.397458 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.397429    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:11:09.488026 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.488001    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:11:09.513834 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.513805    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls\") pod \"251c1551-b708-4f55-a230-5578c7409efc\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") "
Jul 31 15:11:09.513991 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.513911    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54wwh\" (UniqueName: \"kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh\") pod \"251c1551-b708-4f55-a230-5578c7409efc\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") "
Jul 31 15:11:09.513991 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.513943    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config\") pod \"251c1551-b708-4f55-a230-5578c7409efc\" (UID: \"251c1551-b708-4f55-a230-5578c7409efc\") "
Jul 31 15:11:09.514474 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.514437    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "251c1551-b708-4f55-a230-5578c7409efc" (UID: "251c1551-b708-4f55-a230-5578c7409efc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:09.516133 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.516111    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "251c1551-b708-4f55-a230-5578c7409efc" (UID: "251c1551-b708-4f55-a230-5578c7409efc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:09.516133 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.516116    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh" (OuterVolumeSpecName: "kube-api-access-54wwh") pod "251c1551-b708-4f55-a230-5578c7409efc" (UID: "251c1551-b708-4f55-a230-5578c7409efc"). InnerVolumeSpecName "kube-api-access-54wwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:09.615004 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.614946    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfg66\" (UniqueName: \"kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66\") pod \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") "
Jul 31 15:11:09.615004 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615004    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config\") pod \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") "
Jul 31 15:11:09.615496 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615060    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls\") pod \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\" (UID: \"7b69d458-b631-4a3c-a66f-f7f5b63d59b3\") "
Jul 31 15:11:09.615496 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615285    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54wwh\" (UniqueName: \"kubernetes.io/projected/251c1551-b708-4f55-a230-5578c7409efc-kube-api-access-54wwh\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:09.615496 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615304    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/251c1551-b708-4f55-a230-5578c7409efc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:09.615496 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615319    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/251c1551-b708-4f55-a230-5578c7409efc-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:09.615496 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.615425    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b69d458-b631-4a3c-a66f-f7f5b63d59b3" (UID: "7b69d458-b631-4a3c-a66f-f7f5b63d59b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:09.617324 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.617299    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b69d458-b631-4a3c-a66f-f7f5b63d59b3" (UID: "7b69d458-b631-4a3c-a66f-f7f5b63d59b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:09.617427 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.617397    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66" (OuterVolumeSpecName: "kube-api-access-mfg66") pod "7b69d458-b631-4a3c-a66f-f7f5b63d59b3" (UID: "7b69d458-b631-4a3c-a66f-f7f5b63d59b3"). InnerVolumeSpecName "kube-api-access-mfg66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:09.648366 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.648333    2568 generic.go:356] "Generic (PLEG): container finished" podID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerID="727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1" exitCode=0
Jul 31 15:11:09.648507 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.648386    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"
Jul 31 15:11:09.648507 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.648416    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerDied","Data":"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"}
Jul 31 15:11:09.648507 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.648481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq" event={"ID":"7b69d458-b631-4a3c-a66f-f7f5b63d59b3","Type":"ContainerDied","Data":"d495ede97cac079023e0465d7a393dfa326715271bc80dd9b5e91759143a8bcb"}
Jul 31 15:11:09.648684 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.648517    2568 scope.go:117] "RemoveContainer" containerID="e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"
Jul 31 15:11:09.650079 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.650056    2568 generic.go:356] "Generic (PLEG): container finished" podID="251c1551-b708-4f55-a230-5578c7409efc" containerID="57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41" exitCode=0
Jul 31 15:11:09.650079 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.650114    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"
Jul 31 15:11:09.650315 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.650133    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerDied","Data":"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"}
Jul 31 15:11:09.650315 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.650186    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g" event={"ID":"251c1551-b708-4f55-a230-5578c7409efc","Type":"ContainerDied","Data":"e7f3b6e1c3e3dfcca67e958b975697ad09a5a1968d6eed36c441e24942ca1a63"}
Jul 31 15:11:09.657185 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.657017    2568 scope.go:117] "RemoveContainer" containerID="727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"
Jul 31 15:11:09.664838 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.664822    2568 scope.go:117] "RemoveContainer" containerID="e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"
Jul 31 15:11:09.665124 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:09.665106    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986\": container with ID starting with e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986 not found: ID does not exist" containerID="e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"
Jul 31 15:11:09.665263 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.665133    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986"} err="failed to get container status \"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986\": rpc error: code = NotFound desc = could not find container \"e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986\": container with ID starting with e0fbef03c11d4377874876b475fa182ab48798b66e8e99b63ea370a853435986 not found: ID does not exist"
Jul 31 15:11:09.665263 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.665153    2568 scope.go:117] "RemoveContainer" containerID="727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"
Jul 31 15:11:09.665453 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:09.665435    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1\": container with ID starting with 727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1 not found: ID does not exist" containerID="727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"
Jul 31 15:11:09.665504 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.665460    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1"} err="failed to get container status \"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1\": rpc error: code = NotFound desc = could not find container \"727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1\": container with ID starting with 727d0a79ac585f22e41d2a020ae400d84f17823482ce68a4f9d5645baf1899d1 not found: ID does not exist"
Jul 31 15:11:09.665504 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.665484    2568 scope.go:117] "RemoveContainer" containerID="1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"
Jul 31 15:11:09.673031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.673013    2568 scope.go:117] "RemoveContainer" containerID="57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"
Jul 31 15:11:09.677215 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.677173    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:11:09.679837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.679817    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-49957-predictor-c6d6b9746-75srq"]
Jul 31 15:11:09.681384 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.681363    2568 scope.go:117] "RemoveContainer" containerID="1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"
Jul 31 15:11:09.681707 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:09.681683    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67\": container with ID starting with 1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67 not found: ID does not exist" containerID="1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"
Jul 31 15:11:09.681789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.681719    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67"} err="failed to get container status \"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67\": rpc error: code = NotFound desc = could not find container \"1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67\": container with ID starting with 1a31441be53325c5254816984ef095415ee674d84b12675621c1636919b5ac67 not found: ID does not exist"
Jul 31 15:11:09.681789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.681745    2568 scope.go:117] "RemoveContainer" containerID="57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"
Jul 31 15:11:09.682094 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:11:09.682071    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41\": container with ID starting with 57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41 not found: ID does not exist" containerID="57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"
Jul 31 15:11:09.682137 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.682101    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41"} err="failed to get container status \"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41\": rpc error: code = NotFound desc = could not find container \"57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41\": container with ID starting with 57e784d8ea9ec25a5ca5abb5d064431ad34ec1b2fa903edbdfc587ccfb579d41 not found: ID does not exist"
Jul 31 15:11:09.694256 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.694234    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:11:09.703268 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.703231    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49957-predictor-bc66879c5-mgq5g"]
Jul 31 15:11:09.716344 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.716314    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfg66\" (UniqueName: \"kubernetes.io/projected/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-api-access-mfg66\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:09.716344 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.716342    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:09.716523 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:09.716353    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b69d458-b631-4a3c-a66f-f7f5b63d59b3-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:11:10.589644 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:10.589551    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="251c1551-b708-4f55-a230-5578c7409efc" path="/var/lib/kubelet/pods/251c1551-b708-4f55-a230-5578c7409efc/volumes"
Jul 31 15:11:10.590053 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:10.590031    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" path="/var/lib/kubelet/pods/7b69d458-b631-4a3c-a66f-f7f5b63d59b3/volumes"
Jul 31 15:11:12.643142 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:12.643113    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:11:12.643650 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:12.643190    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:11:12.643650 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:12.643600    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:12.643650 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:12.643644    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:18.570754 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:18.570718    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:11:18.571177 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:18.570720    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:11:22.644210 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:22.644155    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:22.644210 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:22.644162    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:28.570764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:28.570718    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:11:28.571365 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:28.570720    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:11:32.643850 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:32.643809    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:32.644265 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:32.643809    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:38.570389 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:38.570350    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:11:38.570823 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:38.570586    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:11:42.644087 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:42.644049    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:42.644087 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:42.644075    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:11:48.571351 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:48.571323    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:11:48.571770 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:48.571405    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:11:52.643848 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:52.643809    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:11:52.644378 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:11:52.644311    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:12:02.644855 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:02.644772    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:12:15.266061 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.266029    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:12:15.266503 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.266289    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" containerID="cri-o://ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931" gracePeriod=30
Jul 31 15:12:15.266503 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.266336    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kube-rbac-proxy" containerID="cri-o://98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56" gracePeriod=30
Jul 31 15:12:15.315890 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.315859    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:12:15.316216 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316188    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kube-rbac-proxy"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316217    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kube-rbac-proxy"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316228    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kube-rbac-proxy"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316234    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kube-rbac-proxy"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316245    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316255    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316267    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container"
Jul 31 15:12:15.316271 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316273    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container"
Jul 31 15:12:15.316560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316324    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kserve-container"
Jul 31 15:12:15.316560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316336    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kserve-container"
Jul 31 15:12:15.316560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316342    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b69d458-b631-4a3c-a66f-f7f5b63d59b3" containerName="kube-rbac-proxy"
Jul 31 15:12:15.316560 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.316349    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="251c1551-b708-4f55-a230-5578c7409efc" containerName="kube-rbac-proxy"
Jul 31 15:12:15.320878 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.320857    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:12:15.321003 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.320968    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.321064 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.321045    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" containerID="cri-o://6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b" gracePeriod=30
Jul 31 15:12:15.321145 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.321097    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kube-rbac-proxy" containerID="cri-o://3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9" gracePeriod=30
Jul 31 15:12:15.323400 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.323379    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b982-kube-rbac-proxy-sar-config\""
Jul 31 15:12:15.323545 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.323379    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7b982-predictor-serving-cert\""
Jul 31 15:12:15.331426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.331401    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:12:15.427056 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.427000    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:12:15.430446 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.430422    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.430622 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.430598    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkk7b\" (UniqueName: \"kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.430714 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.430658    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.430776 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.430724    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.433338 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.433317    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b982-predictor-serving-cert\""
Jul 31 15:12:15.433338 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.433326    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b982-kube-rbac-proxy-sar-config\""
Jul 31 15:12:15.443410 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.443390    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:12:15.531385 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531300    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmtz\" (UniqueName: \"kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.531385 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531340    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkk7b\" (UniqueName: \"kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.531385 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531377    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.531635 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531393    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.531635 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531416    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.531635 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.531635 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:15.531510    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7b982-predictor-serving-cert: secret "success-200-isvc-7b982-predictor-serving-cert" not found
Jul 31 15:12:15.531635 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:15.531576    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls podName:5afd39db-ff97-48a5-85d6-9b9266fa58b2 nodeName:}" failed. No retries permitted until 2026-07-31 15:12:16.031555225 +0000 UTC m=+1447.985869108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls") pod "success-200-isvc-7b982-predictor-5857f8478-6d7w5" (UID: "5afd39db-ff97-48a5-85d6-9b9266fa58b2") : secret "success-200-isvc-7b982-predictor-serving-cert" not found
Jul 31 15:12:15.532003 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.531985    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.541150 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.541121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkk7b\" (UniqueName: \"kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:15.632642 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.632604    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmtz\" (UniqueName: \"kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.632826 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.632687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.632826 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.632720    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.633387 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.633367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.635296 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.635275    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.641732 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.641708    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmtz\" (UniqueName: \"kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz\") pod \"error-404-isvc-7b982-predictor-d56595d99-q8npg\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") " pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.740913 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.740866    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:15.876471 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.876442    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:12:15.879752 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:12:15.879726    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420b74ef_0f55_4f6f_a974_beff045533bb.slice/crio-b506c2ee8c85c6fbed276bc04ac3be5078a67179aaaf939d3a484e4810e77770 WatchSource:0}: Error finding container b506c2ee8c85c6fbed276bc04ac3be5078a67179aaaf939d3a484e4810e77770: Status 404 returned error can't find the container with id b506c2ee8c85c6fbed276bc04ac3be5078a67179aaaf939d3a484e4810e77770
Jul 31 15:12:15.909816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.909780    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerStarted","Data":"b506c2ee8c85c6fbed276bc04ac3be5078a67179aaaf939d3a484e4810e77770"}
Jul 31 15:12:15.911911 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.911887    2568 generic.go:356] "Generic (PLEG): container finished" podID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerID="98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56" exitCode=2
Jul 31 15:12:15.912099 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.911956    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerDied","Data":"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"}
Jul 31 15:12:15.913705 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.913685    2568 generic.go:356] "Generic (PLEG): container finished" podID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerID="3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9" exitCode=2
Jul 31 15:12:15.913819 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:15.913731    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerDied","Data":"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"}
Jul 31 15:12:16.036436 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.036401    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:16.038952 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.038932    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") pod \"success-200-isvc-7b982-predictor-5857f8478-6d7w5\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") " pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:16.239060 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.239020    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:16.377624 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.377578    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:12:16.380090 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:12:16.380063    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afd39db_ff97_48a5_85d6_9b9266fa58b2.slice/crio-d12958e70ecee924dd9f358d241588ed24e8c78cd1dfb19d24117679a5715108 WatchSource:0}: Error finding container d12958e70ecee924dd9f358d241588ed24e8c78cd1dfb19d24117679a5715108: Status 404 returned error can't find the container with id d12958e70ecee924dd9f358d241588ed24e8c78cd1dfb19d24117679a5715108
Jul 31 15:12:16.919355 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.919251    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerStarted","Data":"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"}
Jul 31 15:12:16.919355 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.919328    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:16.919597 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.919357    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerStarted","Data":"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"}
Jul 31 15:12:16.919597 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.919388    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:16.920664 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.920632    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:16.921209 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.921163    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerStarted","Data":"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"}
Jul 31 15:12:16.921302 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.921247    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerStarted","Data":"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"}
Jul 31 15:12:16.921302 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.921285    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:16.921376 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.921314    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:16.921376 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.921340    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerStarted","Data":"d12958e70ecee924dd9f358d241588ed24e8c78cd1dfb19d24117679a5715108"}
Jul 31 15:12:16.922288 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.922267    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:12:16.938815 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.938770    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podStartSLOduration=1.938757826 podStartE2EDuration="1.938757826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:12:16.937597806 +0000 UTC m=+1448.891911713" watchObservedRunningTime="2026-07-31 15:12:16.938757826 +0000 UTC m=+1448.893071732"
Jul 31 15:12:16.956156 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:16.956116    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podStartSLOduration=1.956102081 podStartE2EDuration="1.956102081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:12:16.954907672 +0000 UTC m=+1448.909221580" watchObservedRunningTime="2026-07-31 15:12:16.956102081 +0000 UTC m=+1448.910416048"
Jul 31 15:12:17.925162 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:17.925115    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:12:17.925581 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:17.925185    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:18.566676 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.566636    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:12:18.566869 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.566635    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 31 15:12:18.571099 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.571070    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:12:18.571099 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.571087    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 31 15:12:18.845074 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.845048    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:12:18.930610 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.930571    2568 generic.go:356] "Generic (PLEG): container finished" podID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerID="ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931" exitCode=0
Jul 31 15:12:18.930980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.930619    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"
Jul 31 15:12:18.930980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.930656    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerDied","Data":"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"}
Jul 31 15:12:18.930980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.930720    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5" event={"ID":"69745f64-c3d6-4e6f-9780-54ca490f1ee2","Type":"ContainerDied","Data":"1c5f98e8e04eb76c91f5119f765c80cf36d2e8028bbabe68bcc7d985dd1f7aeb"}
Jul 31 15:12:18.930980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.930748    2568 scope.go:117] "RemoveContainer" containerID="98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"
Jul 31 15:12:18.938900 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.938886    2568 scope.go:117] "RemoveContainer" containerID="ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"
Jul 31 15:12:18.945913 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.945900    2568 scope.go:117] "RemoveContainer" containerID="98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"
Jul 31 15:12:18.946137 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:18.946117    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56\": container with ID starting with 98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56 not found: ID does not exist" containerID="98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"
Jul 31 15:12:18.946231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.946149    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56"} err="failed to get container status \"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56\": rpc error: code = NotFound desc = could not find container \"98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56\": container with ID starting with 98bcf99a1b070e2e2a06191abfe381fe6e268a82f27ecd40e302df1496a6ed56 not found: ID does not exist"
Jul 31 15:12:18.946231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.946174    2568 scope.go:117] "RemoveContainer" containerID="ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"
Jul 31 15:12:18.946422 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:18.946405    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931\": container with ID starting with ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931 not found: ID does not exist" containerID="ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"
Jul 31 15:12:18.946464 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.946430    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931"} err="failed to get container status \"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931\": rpc error: code = NotFound desc = could not find container \"ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931\": container with ID starting with ffeb5835b786ec79367b0118fe32a16fa3ad632e77f6827c8b46234311cbb931 not found: ID does not exist"
Jul 31 15:12:18.960203 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.960171    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z64km\" (UniqueName: \"kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km\") pod \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") "
Jul 31 15:12:18.960312 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.960296    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config\") pod \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") "
Jul 31 15:12:18.960373 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.960324    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") pod \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\" (UID: \"69745f64-c3d6-4e6f-9780-54ca490f1ee2\") "
Jul 31 15:12:18.960628 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.960610    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69745f64-c3d6-4e6f-9780-54ca490f1ee2" (UID: "69745f64-c3d6-4e6f-9780-54ca490f1ee2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:12:18.962374 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.962355    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km" (OuterVolumeSpecName: "kube-api-access-z64km") pod "69745f64-c3d6-4e6f-9780-54ca490f1ee2" (UID: "69745f64-c3d6-4e6f-9780-54ca490f1ee2"). InnerVolumeSpecName "kube-api-access-z64km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:12:18.962448 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:18.962431    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69745f64-c3d6-4e6f-9780-54ca490f1ee2" (UID: "69745f64-c3d6-4e6f-9780-54ca490f1ee2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:12:19.061517 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.061484    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.061517 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.061514    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69745f64-c3d6-4e6f-9780-54ca490f1ee2-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.061517 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.061528    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64km\" (UniqueName: \"kubernetes.io/projected/69745f64-c3d6-4e6f-9780-54ca490f1ee2-kube-api-access-z64km\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.253133 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.253101    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:12:19.257864 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.257839    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a1442-predictor-5759c7c998-p46m5"]
Jul 31 15:12:19.452452 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.452426    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:12:19.565097 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.565014    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config\") pod \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") "
Jul 31 15:12:19.565097 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.565077    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrrr\" (UniqueName: \"kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr\") pod \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") "
Jul 31 15:12:19.565369 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.565110    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls\") pod \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\" (UID: \"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0\") "
Jul 31 15:12:19.565433 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.565407    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" (UID: "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:12:19.567309 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.567287    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" (UID: "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:12:19.567381 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.567348    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr" (OuterVolumeSpecName: "kube-api-access-kmrrr") pod "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" (UID: "61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0"). InnerVolumeSpecName "kube-api-access-kmrrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:12:19.666031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.665990    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.666031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.666024    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrrr\" (UniqueName: \"kubernetes.io/projected/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-kube-api-access-kmrrr\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.666031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.666034    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:12:19.936258 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.936219    2568 generic.go:356] "Generic (PLEG): container finished" podID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerID="6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b" exitCode=0
Jul 31 15:12:19.936657 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.936269    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerDied","Data":"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"}
Jul 31 15:12:19.936657 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.936283    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"
Jul 31 15:12:19.936657 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.936329    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh" event={"ID":"61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0","Type":"ContainerDied","Data":"cc73f6af915a5d3c4ea8db34b8c32b778b369ab5687f7ef41a4b84fe7a0347f3"}
Jul 31 15:12:19.936657 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.936367    2568 scope.go:117] "RemoveContainer" containerID="3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"
Jul 31 15:12:19.945056 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.945036    2568 scope.go:117] "RemoveContainer" containerID="6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"
Jul 31 15:12:19.952276 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.952257    2568 scope.go:117] "RemoveContainer" containerID="3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"
Jul 31 15:12:19.952513 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:19.952494    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9\": container with ID starting with 3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9 not found: ID does not exist" containerID="3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"
Jul 31 15:12:19.952585 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.952524    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9"} err="failed to get container status \"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9\": rpc error: code = NotFound desc = could not find container \"3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9\": container with ID starting with 3396ab60ce851964e1bccf981cebb2b8182b0ea1102b2eadb038060725e621e9 not found: ID does not exist"
Jul 31 15:12:19.952585 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.952546    2568 scope.go:117] "RemoveContainer" containerID="6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"
Jul 31 15:12:19.952781 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:12:19.952766    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b\": container with ID starting with 6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b not found: ID does not exist" containerID="6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"
Jul 31 15:12:19.952833 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.952791    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b"} err="failed to get container status \"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b\": rpc error: code = NotFound desc = could not find container \"6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b\": container with ID starting with 6b6036b346311903c4c98ce9f7ad7fabe26b73769b1bdf1107bf946c3edaee8b not found: ID does not exist"
Jul 31 15:12:19.957948 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.957924    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:12:19.964293 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:19.964275    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a1442-predictor-84bf5c887f-bmcrh"]
Jul 31 15:12:20.588718 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:20.588687    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" path="/var/lib/kubelet/pods/61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0/volumes"
Jul 31 15:12:20.589086 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:20.589074    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" path="/var/lib/kubelet/pods/69745f64-c3d6-4e6f-9780-54ca490f1ee2/volumes"
Jul 31 15:12:22.930173 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:22.930135    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:12:22.930648 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:22.930573    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:12:22.930648 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:22.930616    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:22.931073 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:22.931048    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:12:32.931105 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:32.931065    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:12:32.931515 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:32.931066    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:42.930837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:42.930800    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:42.931331 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:42.931024    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:12:52.930656 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:52.930617    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:12:52.931130 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:12:52.931102    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:13:02.931373 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:13:02.931341    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:13:02.931825 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:13:02.931710    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:13:08.521162 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:13:08.521133    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:13:08.522916 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:13:08.522893    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:18:08.544105 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:18:08.544070    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:18:08.545950 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:18:08.545929    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:20:20.170655 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.170623    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:20:20.173356 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.170993    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" containerID="cri-o://1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8" gracePeriod=30
Jul 31 15:20:20.173356 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.171032    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kube-rbac-proxy" containerID="cri-o://8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6" gracePeriod=30
Jul 31 15:20:20.229029 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.228982    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:20:20.229311 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229298    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kube-rbac-proxy"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229313    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kube-rbac-proxy"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229327    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229333    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229346    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229351    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229360    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kube-rbac-proxy"
Jul 31 15:20:20.229367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229365    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kube-rbac-proxy"
Jul 31 15:20:20.229601 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229414    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kube-rbac-proxy"
Jul 31 15:20:20.229601 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229421    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kserve-container"
Jul 31 15:20:20.229601 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229428    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ffb3d0-d0f8-4504-b0a0-b5d16b80a0c0" containerName="kserve-container"
Jul 31 15:20:20.229601 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.229433    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="69745f64-c3d6-4e6f-9780-54ca490f1ee2" containerName="kube-rbac-proxy"
Jul 31 15:20:20.232288 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.232267    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.234767 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.234747    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e31d5-kube-rbac-proxy-sar-config\""
Jul 31 15:20:20.234892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.234795    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e31d5-predictor-serving-cert\""
Jul 31 15:20:20.244665 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.244640    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:20:20.306140 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.306106    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:20:20.309402 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.309387    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.311867 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.311844    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e31d5-kube-rbac-proxy-sar-config\""
Jul 31 15:20:20.311982 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.311951    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e31d5-predictor-serving-cert\""
Jul 31 15:20:20.320515 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.320492    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:20:20.329187 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.329160    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.329318 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.329300    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksvc\" (UniqueName: \"kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.329364 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.329341    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.359798 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.359768    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:20:20.360013 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.359987    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" containerID="cri-o://233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b" gracePeriod=30
Jul 31 15:20:20.360098 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.360016    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kube-rbac-proxy" containerID="cri-o://d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8" gracePeriod=30
Jul 31 15:20:20.430592 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430502    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.430592 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430541    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.430837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430602    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmppw\" (UniqueName: \"kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.430837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksvc\" (UniqueName: \"kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.430837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430645    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.430837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.430673    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.431177 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.431156    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.433124 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.433105    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.439907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.439883    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksvc\" (UniqueName: \"kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc\") pod \"success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") " pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.531085 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.531055    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmppw\" (UniqueName: \"kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.531262 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.531122    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.531262 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.531161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.531392 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:20.531283    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-serving-cert: secret "error-404-isvc-e31d5-predictor-serving-cert" not found
Jul 31 15:20:20.531392 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:20.531343    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls podName:5588be0d-175b-4bc7-a25e-91e4bfb55df4 nodeName:}" failed. No retries permitted until 2026-07-31 15:20:21.031323108 +0000 UTC m=+1932.985636991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls") pod "error-404-isvc-e31d5-predictor-8545984775-n6bpj" (UID: "5588be0d-175b-4bc7-a25e-91e4bfb55df4") : secret "error-404-isvc-e31d5-predictor-serving-cert" not found
Jul 31 15:20:20.531840 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.531815    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.542308 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.542277    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmppw\" (UniqueName: \"kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:20.542452 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.542438    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:20.676669 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.676636    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:20:20.679821 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:20:20.679794    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90a334f_7eea_41c6_aeaa_a44c41f481f3.slice/crio-471b1075d1e45ea5ad1355bc861f904237ec86fd5d174cad47078c2eb9ba0567 WatchSource:0}: Error finding container 471b1075d1e45ea5ad1355bc861f904237ec86fd5d174cad47078c2eb9ba0567: Status 404 returned error can't find the container with id 471b1075d1e45ea5ad1355bc861f904237ec86fd5d174cad47078c2eb9ba0567
Jul 31 15:20:20.712570 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.712539    2568 generic.go:356] "Generic (PLEG): container finished" podID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerID="d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8" exitCode=2
Jul 31 15:20:20.712697 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.712605    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerDied","Data":"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"}
Jul 31 15:20:20.713898 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.713847    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerStarted","Data":"471b1075d1e45ea5ad1355bc861f904237ec86fd5d174cad47078c2eb9ba0567"}
Jul 31 15:20:20.715656 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.715637    2568 generic.go:356] "Generic (PLEG): container finished" podID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerID="8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6" exitCode=2
Jul 31 15:20:20.715751 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:20.715681    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerDied","Data":"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"}
Jul 31 15:20:21.036333 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.036248    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:21.038784 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.038764    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") pod \"error-404-isvc-e31d5-predictor-8545984775-n6bpj\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") " pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:21.219885 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.219844    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:21.353503 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.353464    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:20:21.357048 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:20:21.357023    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5588be0d_175b_4bc7_a25e_91e4bfb55df4.slice/crio-5017af668f137c99030bd6a3f5f7ce2443526cad218c5f41d68868ff59cb2751 WatchSource:0}: Error finding container 5017af668f137c99030bd6a3f5f7ce2443526cad218c5f41d68868ff59cb2751: Status 404 returned error can't find the container with id 5017af668f137c99030bd6a3f5f7ce2443526cad218c5f41d68868ff59cb2751
Jul 31 15:20:21.721430 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.721384    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerStarted","Data":"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"}
Jul 31 15:20:21.721644 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.721495    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerStarted","Data":"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"}
Jul 31 15:20:21.721644 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.721535    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:21.721644 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.721567    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:21.721644 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.721593    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerStarted","Data":"5017af668f137c99030bd6a3f5f7ce2443526cad218c5f41d68868ff59cb2751"}
Jul 31 15:20:21.723280 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.723244    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:20:21.723404 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.723327    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerStarted","Data":"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"}
Jul 31 15:20:21.723404 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.723372    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerStarted","Data":"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"}
Jul 31 15:20:21.723534 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.723523    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:21.723589 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.723546    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:21.724501 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.724477    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:21.739878 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.739831    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podStartSLOduration=1.7398163709999999 podStartE2EDuration="1.739816371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:21.738363665 +0000 UTC m=+1933.692677572" watchObservedRunningTime="2026-07-31 15:20:21.739816371 +0000 UTC m=+1933.694130276"
Jul 31 15:20:21.757093 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:21.757043    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podStartSLOduration=1.757028149 podStartE2EDuration="1.757028149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:21.756081867 +0000 UTC m=+1933.710395774" watchObservedRunningTime="2026-07-31 15:20:21.757028149 +0000 UTC m=+1933.711342099"
Jul 31 15:20:22.639054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.639006    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 31 15:20:22.639509 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.639008    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 31 15:20:22.644489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.644461    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:20:22.644611 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.644463    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 31 15:20:22.727861 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.727817    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:22.728047 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:22.727914    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:20:23.624485 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.624460    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:20:23.732563 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.732530    2568 generic.go:356] "Generic (PLEG): container finished" podID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerID="1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8" exitCode=0
Jul 31 15:20:23.732933 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.732580    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"
Jul 31 15:20:23.732933 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.732614    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerDied","Data":"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"}
Jul 31 15:20:23.732933 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.732672    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296" event={"ID":"5350a19f-b95b-4b88-afbb-ffa9574c5eae","Type":"ContainerDied","Data":"767f139507ae19ee2f3bc38676734e094aa0e5bf53e54d0eb7dff9b79e70c26e"}
Jul 31 15:20:23.732933 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.732703    2568 scope.go:117] "RemoveContainer" containerID="8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"
Jul 31 15:20:23.757888 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.757860    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config\") pod \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") "
Jul 31 15:20:23.758024 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.757913    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") pod \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") "
Jul 31 15:20:23.758024 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.757960    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h2s2\" (UniqueName: \"kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2\") pod \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\" (UID: \"5350a19f-b95b-4b88-afbb-ffa9574c5eae\") "
Jul 31 15:20:23.758287 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.758268    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5350a19f-b95b-4b88-afbb-ffa9574c5eae" (UID: "5350a19f-b95b-4b88-afbb-ffa9574c5eae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:20:23.760164 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.760141    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5350a19f-b95b-4b88-afbb-ffa9574c5eae" (UID: "5350a19f-b95b-4b88-afbb-ffa9574c5eae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:20:23.760265 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.760170    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2" (OuterVolumeSpecName: "kube-api-access-9h2s2") pod "5350a19f-b95b-4b88-afbb-ffa9574c5eae" (UID: "5350a19f-b95b-4b88-afbb-ffa9574c5eae"). InnerVolumeSpecName "kube-api-access-9h2s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:20:23.772562 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.772536    2568 scope.go:117] "RemoveContainer" containerID="1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"
Jul 31 15:20:23.780277 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.780254    2568 scope.go:117] "RemoveContainer" containerID="8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"
Jul 31 15:20:23.780622 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:23.780597    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6\": container with ID starting with 8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6 not found: ID does not exist" containerID="8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"
Jul 31 15:20:23.780726 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.780630    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6"} err="failed to get container status \"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6\": rpc error: code = NotFound desc = could not find container \"8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6\": container with ID starting with 8735a185734b36c686f6949374784fbdc21544097af6d77c1ce8fa044f790bc6 not found: ID does not exist"
Jul 31 15:20:23.780726 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.780649    2568 scope.go:117] "RemoveContainer" containerID="1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"
Jul 31 15:20:23.780917 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:23.780896    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8\": container with ID starting with 1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8 not found: ID does not exist" containerID="1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"
Jul 31 15:20:23.780967 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.780923    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8"} err="failed to get container status \"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8\": rpc error: code = NotFound desc = could not find container \"1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8\": container with ID starting with 1f9c976bd1983a52c630e1a75711713257b9e22a38f27dbf2fde8ddfa17d23e8 not found: ID does not exist"
Jul 31 15:20:23.858892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.858863    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2s2\" (UniqueName: \"kubernetes.io/projected/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-api-access-9h2s2\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:23.858892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.858890    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5350a19f-b95b-4b88-afbb-ffa9574c5eae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:23.859100 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.858900    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5350a19f-b95b-4b88-afbb-ffa9574c5eae-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:23.902746 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:23.902715    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:20:24.053659 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.053628    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:20:24.057742 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.057716    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-06a6c-predictor-69d6f4dc94-zf296"]
Jul 31 15:20:24.060420 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.060404    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qkph\" (UniqueName: \"kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph\") pod \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") "
Jul 31 15:20:24.060488 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.060465    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config\") pod \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") "
Jul 31 15:20:24.060527 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.060507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") pod \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\" (UID: \"e3a6010f-eee8-4a8c-894e-d5b21bec7a31\") "
Jul 31 15:20:24.060803 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.060772    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3a6010f-eee8-4a8c-894e-d5b21bec7a31" (UID: "e3a6010f-eee8-4a8c-894e-d5b21bec7a31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:20:24.062714 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.062691    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3a6010f-eee8-4a8c-894e-d5b21bec7a31" (UID: "e3a6010f-eee8-4a8c-894e-d5b21bec7a31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:20:24.062789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.062725    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph" (OuterVolumeSpecName: "kube-api-access-9qkph") pod "e3a6010f-eee8-4a8c-894e-d5b21bec7a31" (UID: "e3a6010f-eee8-4a8c-894e-d5b21bec7a31"). InnerVolumeSpecName "kube-api-access-9qkph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:20:24.161553 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.161512    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkph\" (UniqueName: \"kubernetes.io/projected/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-api-access-9qkph\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:24.161553 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.161548    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:24.161553 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.161562    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3a6010f-eee8-4a8c-894e-d5b21bec7a31-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:20:24.590426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.590392    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" path="/var/lib/kubelet/pods/5350a19f-b95b-4b88-afbb-ffa9574c5eae/volumes"
Jul 31 15:20:24.738573 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.738532    2568 generic.go:356] "Generic (PLEG): container finished" podID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerID="233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b" exitCode=0
Jul 31 15:20:24.739026 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.738589    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"
Jul 31 15:20:24.739026 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.738593    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerDied","Data":"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"}
Jul 31 15:20:24.739026 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.738714    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g" event={"ID":"e3a6010f-eee8-4a8c-894e-d5b21bec7a31","Type":"ContainerDied","Data":"f36dcb245c03120b7d47b7b1e4989fc6b5258f0b732d4f62b00d0afe6a3cb64f"}
Jul 31 15:20:24.739026 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.738746    2568 scope.go:117] "RemoveContainer" containerID="d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"
Jul 31 15:20:24.746544 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.746526    2568 scope.go:117] "RemoveContainer" containerID="233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"
Jul 31 15:20:24.753975 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.753947    2568 scope.go:117] "RemoveContainer" containerID="d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"
Jul 31 15:20:24.754247 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:24.754224    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8\": container with ID starting with d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8 not found: ID does not exist" containerID="d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"
Jul 31 15:20:24.754288 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.754260    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8"} err="failed to get container status \"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8\": rpc error: code = NotFound desc = could not find container \"d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8\": container with ID starting with d47bf217160beb1d6ffaaa681c518a47fe9f795b8b179100a2aee32dc0c237a8 not found: ID does not exist"
Jul 31 15:20:24.754288 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.754284    2568 scope.go:117] "RemoveContainer" containerID="233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"
Jul 31 15:20:24.754527 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:20:24.754511    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b\": container with ID starting with 233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b not found: ID does not exist" containerID="233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"
Jul 31 15:20:24.754563 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.754539    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b"} err="failed to get container status \"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b\": rpc error: code = NotFound desc = could not find container \"233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b\": container with ID starting with 233ac5665c499488411ef27a5bde977f849d6b3fa0724143de86e55ca316b21b not found: ID does not exist"
Jul 31 15:20:24.757230 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.757184    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:20:24.762019 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:24.761997    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-06a6c-predictor-5766f5fbbc-h8s7g"]
Jul 31 15:20:26.589556 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:26.589507    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" path="/var/lib/kubelet/pods/e3a6010f-eee8-4a8c-894e-d5b21bec7a31/volumes"
Jul 31 15:20:27.733179 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:27.732541    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:20:27.733647 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:27.733461    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:20:27.733647 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:27.733636    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:20:27.734106 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:27.734080    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:37.734247 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:37.734189    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:20:37.734723 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:37.734216    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:47.733482 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:47.733441    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:20:47.734658 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:47.734634    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:57.734121 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:57.734082    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:20:57.734550 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:20:57.734083    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:21:07.734501 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:07.734414    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:21:07.734977 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:07.734960    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:21:30.267704 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.267625    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:21:30.268390 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.267852    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" containerID="cri-o://0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664" gracePeriod=30
Jul 31 15:21:30.268390 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.267890    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kube-rbac-proxy" containerID="cri-o://7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679" gracePeriod=30
Jul 31 15:21:30.326301 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.326268    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:21:30.326505 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.326478    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" containerID="cri-o://eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af" gracePeriod=30
Jul 31 15:21:30.326575 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.326535    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kube-rbac-proxy" containerID="cri-o://9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48" gracePeriod=30
Jul 31 15:21:30.342660 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342623    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:21:30.342944 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342930    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container"
Jul 31 15:21:30.342944 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342943    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342953    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342959    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342967    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342972    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342983    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.342989    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.343031    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.343039    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kserve-container"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.343047    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a6010f-eee8-4a8c-894e-d5b21bec7a31" containerName="kube-rbac-proxy"
Jul 31 15:21:30.343054 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.343054    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5350a19f-b95b-4b88-afbb-ffa9574c5eae" containerName="kserve-container"
Jul 31 15:21:30.346182 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.346166    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.350013 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.349983    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7902a-kube-rbac-proxy-sar-config\""
Jul 31 15:21:30.350124 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.349983    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7902a-predictor-serving-cert\""
Jul 31 15:21:30.365986 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.365954    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:21:30.476597 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.476555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97q2f\" (UniqueName: \"kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.476766 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.476626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.476766 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.476719    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.493688 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.493656    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:21:30.497226 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.497184    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.503367 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.503314    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7902a-kube-rbac-proxy-sar-config\""
Jul 31 15:21:30.503550 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.503376    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7902a-predictor-serving-cert\""
Jul 31 15:21:30.523600 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.523521    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:21:30.578013 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.577972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.578184 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578031    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.578184 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.578300 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578208    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.578300 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97q2f\" (UniqueName: \"kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.578415 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578306    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxwj\" (UniqueName: \"kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.578782 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.578761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.580638 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.580620    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.588793 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.588763    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97q2f\" (UniqueName: \"kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f\") pod \"success-200-isvc-7902a-predictor-8dd58f79b-bjhqg\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") " pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.656310 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.656279    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.678651 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.678624    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.678790 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.678687    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxwj\" (UniqueName: \"kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.678790 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.678729    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.679447 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.679413    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.681079 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.681056    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.689347 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.689320    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxwj\" (UniqueName: \"kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj\") pod \"error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") " pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.800667 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.800637    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:21:30.802537 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:21:30.802512    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3460efc7_de17_47da_89e9_87bcba812bbe.slice/crio-771b70b51b5039d11df66f1e69043a2f5f996b69f3a1cd96d98a5eb6b4c989d6 WatchSource:0}: Error finding container 771b70b51b5039d11df66f1e69043a2f5f996b69f3a1cd96d98a5eb6b4c989d6: Status 404 returned error can't find the container with id 771b70b51b5039d11df66f1e69043a2f5f996b69f3a1cd96d98a5eb6b4c989d6
Jul 31 15:21:30.807155 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.807135    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:30.838116 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.838084    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:21:30.843156 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.843132    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:30.845596 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.845576    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-faf8d-predictor-serving-cert\""
Jul 31 15:21:30.845921 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.845890    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-faf8d-kube-rbac-proxy-sar-config\""
Jul 31 15:21:30.852892 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.852862    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:21:30.940841 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.940818    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:21:30.945510 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.945488    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:30.948442 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.948419    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-faf8d-predictor-serving-cert\""
Jul 31 15:21:30.948606 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.948591    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-faf8d-kube-rbac-proxy-sar-config\""
Jul 31 15:21:30.954709 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.954687    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:21:30.970885 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.970861    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:21:30.972794 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:21:30.972771    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c74c8bc_28a4_467a_a807_0e14cfdad979.slice/crio-65c7a4736814015736aafa5e14d47b09adfce0f113918c4808fc753806ecfd14 WatchSource:0}: Error finding container 65c7a4736814015736aafa5e14d47b09adfce0f113918c4808fc753806ecfd14: Status 404 returned error can't find the container with id 65c7a4736814015736aafa5e14d47b09adfce0f113918c4808fc753806ecfd14
Jul 31 15:21:30.980479 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.980454    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:30.980580 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.980504    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:30.980580 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.980566    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4kd\" (UniqueName: \"kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:30.981362 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.981336    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerStarted","Data":"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"}
Jul 31 15:21:30.981471 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.981402    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerStarted","Data":"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"}
Jul 31 15:21:30.981471 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.981440    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.981607 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.981587    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:30.981662 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.981641    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerStarted","Data":"771b70b51b5039d11df66f1e69043a2f5f996b69f3a1cd96d98a5eb6b4c989d6"}
Jul 31 15:21:30.982696 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.982670    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:21:30.983295 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.983268    2568 generic.go:356] "Generic (PLEG): container finished" podID="420b74ef-0f55-4f6f-a974-beff045533bb" containerID="9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48" exitCode=2
Jul 31 15:21:30.983406 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.983338    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerDied","Data":"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"}
Jul 31 15:21:30.984977 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.984955    2568 generic.go:356] "Generic (PLEG): container finished" podID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerID="7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679" exitCode=2
Jul 31 15:21:30.985131 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.985036    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerDied","Data":"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"}
Jul 31 15:21:30.986263 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:30.986242    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerStarted","Data":"65c7a4736814015736aafa5e14d47b09adfce0f113918c4808fc753806ecfd14"}
Jul 31 15:21:31.009933 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.009872    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podStartSLOduration=1.00984945 podStartE2EDuration="1.00984945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:31.007856603 +0000 UTC m=+2002.962170505" watchObservedRunningTime="2026-07-31 15:21:31.00984945 +0000 UTC m=+2002.964163359"
Jul 31 15:21:31.029269 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.029241    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:21:31.029483 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.029458    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container" containerID="cri-o://b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090" gracePeriod=30
Jul 31 15:21:31.029553 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.029497    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kube-rbac-proxy" containerID="cri-o://78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e" gracePeriod=30
Jul 31 15:21:31.065790 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.065759    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:21:31.065999 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.065971    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container" containerID="cri-o://4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154" gracePeriod=30
Jul 31 15:21:31.066112 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.066008    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kube-rbac-proxy" containerID="cri-o://6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e" gracePeriod=30
Jul 31 15:21:31.081050 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081025    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4kd\" (UniqueName: \"kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.081149 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081088    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.081289 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:31.081270    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-serving-cert: secret "success-200-isvc-faf8d-predictor-serving-cert" not found
Jul 31 15:21:31.081358 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081272    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.081358 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:31.081338    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls podName:5b3aad63-8ee8-4381-bffc-2af39bbccc34 nodeName:}" failed. No retries permitted until 2026-07-31 15:21:31.581317303 +0000 UTC m=+2003.535631206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls") pod "success-200-isvc-faf8d-predictor-5896849644-7rtxg" (UID: "5b3aad63-8ee8-4381-bffc-2af39bbccc34") : secret "success-200-isvc-faf8d-predictor-serving-cert" not found
Jul 31 15:21:31.081477 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.081477 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081449    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.081645 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081494    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpvh\" (UniqueName: \"kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.082016 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.081994    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.096410 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.093869    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4kd\" (UniqueName: \"kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.182320 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.182292    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.182422 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.182342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpvh\" (UniqueName: \"kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.182479 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.182420    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.182577 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:31.182557    2568 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-serving-cert: secret "error-404-isvc-faf8d-predictor-serving-cert" not found
Jul 31 15:21:31.182635 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:31.182628    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls podName:5fa64ced-8e39-4ede-9456-34d853453666 nodeName:}" failed. No retries permitted until 2026-07-31 15:21:31.682607282 +0000 UTC m=+2003.636921167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls") pod "error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" (UID: "5fa64ced-8e39-4ede-9456-34d853453666") : secret "error-404-isvc-faf8d-predictor-serving-cert" not found
Jul 31 15:21:31.183013 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.182991    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.192752 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.192726    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpvh\" (UniqueName: \"kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.586763 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.586725    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.589350 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.589330    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") pod \"success-200-isvc-faf8d-predictor-5896849644-7rtxg\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") " pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.687245 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.687183    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.689796 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.689774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") pod \"error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") " pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.763789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.763751    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:31.857632 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.857601    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:31.903306 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.903143    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:21:31.909912 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:21:31.909881    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3aad63_8ee8_4381_bffc_2af39bbccc34.slice/crio-7a7d2ae1c91415321c0f3c31033bab5a322fc16c520908efa16b604226d1194c WatchSource:0}: Error finding container 7a7d2ae1c91415321c0f3c31033bab5a322fc16c520908efa16b604226d1194c: Status 404 returned error can't find the container with id 7a7d2ae1c91415321c0f3c31033bab5a322fc16c520908efa16b604226d1194c
Jul 31 15:21:31.995852 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.995652    2568 generic.go:356] "Generic (PLEG): container finished" podID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerID="78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e" exitCode=2
Jul 31 15:21:31.995852 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.995772    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerDied","Data":"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"}
Jul 31 15:21:31.997980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.997835    2568 generic.go:356] "Generic (PLEG): container finished" podID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerID="6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e" exitCode=2
Jul 31 15:21:31.997980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.997913    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerDied","Data":"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"}
Jul 31 15:21:31.999799 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.999771    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerStarted","Data":"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"}
Jul 31 15:21:31.999896 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:31.999864    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerStarted","Data":"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"}
Jul 31 15:21:32.006584 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.005412    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:21:32.006584 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.006252    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:32.006584 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.006304    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:32.007693 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.007542    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:21:32.009573 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.009550    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerStarted","Data":"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"}
Jul 31 15:21:32.009654 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.009599    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerStarted","Data":"7a7d2ae1c91415321c0f3c31033bab5a322fc16c520908efa16b604226d1194c"}
Jul 31 15:21:32.009972 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.009949    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:21:32.013329 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:21:32.013308    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa64ced_8e39_4ede_9456_34d853453666.slice/crio-e5cb6a34f3b6c80f928cb881da1d4b93a50eb5d3adc364eb7b5904f18e058b82 WatchSource:0}: Error finding container e5cb6a34f3b6c80f928cb881da1d4b93a50eb5d3adc364eb7b5904f18e058b82: Status 404 returned error can't find the container with id e5cb6a34f3b6c80f928cb881da1d4b93a50eb5d3adc364eb7b5904f18e058b82
Jul 31 15:21:32.022423 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.022382    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podStartSLOduration=2.022365793 podStartE2EDuration="2.022365793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:32.02038907 +0000 UTC m=+2003.974702978" watchObservedRunningTime="2026-07-31 15:21:32.022365793 +0000 UTC m=+2003.976679693"
Jul 31 15:21:32.728396 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.728361    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 31 15:21:32.728788 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.728349    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 31 15:21:32.926100 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.926057    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 31 15:21:32.926317 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.926055    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 31 15:21:32.931095 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.931066    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 31 15:21:32.931225 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:32.931069    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 31 15:21:33.016189 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.016089    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerStarted","Data":"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"}
Jul 31 15:21:33.016189 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.016156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerStarted","Data":"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"}
Jul 31 15:21:33.016440 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.016211    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:33.016440 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.016238    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerStarted","Data":"e5cb6a34f3b6c80f928cb881da1d4b93a50eb5d3adc364eb7b5904f18e058b82"}
Jul 31 15:21:33.017807 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.017783    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerStarted","Data":"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"}
Jul 31 15:21:33.018228 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.018182    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:21:33.036517 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.036476    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podStartSLOduration=3.036460373 podStartE2EDuration="3.036460373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:33.035161386 +0000 UTC m=+2004.989475292" watchObservedRunningTime="2026-07-31 15:21:33.036460373 +0000 UTC m=+2004.990774280"
Jul 31 15:21:33.054777 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:33.054686    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podStartSLOduration=3.054666482 podStartE2EDuration="3.054666482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:33.054289877 +0000 UTC m=+2005.008603799" watchObservedRunningTime="2026-07-31 15:21:33.054666482 +0000 UTC m=+2005.008980390"
Jul 31 15:21:34.021670 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.021633    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:34.022149 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.021825    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:21:34.022255 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.022229    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:34.022433 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.022414    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:34.022988 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.022966    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:21:34.023031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.022978    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:21:34.606481 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.606399    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:21:34.716848 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.716810    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") pod \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") "
Jul 31 15:21:34.717043 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.716876    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkk7b\" (UniqueName: \"kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b\") pod \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") "
Jul 31 15:21:34.717043 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.716988    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config\") pod \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\" (UID: \"5afd39db-ff97-48a5-85d6-9b9266fa58b2\") "
Jul 31 15:21:34.717417 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.717390    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5afd39db-ff97-48a5-85d6-9b9266fa58b2" (UID: "5afd39db-ff97-48a5-85d6-9b9266fa58b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:34.719138 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.719106    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5afd39db-ff97-48a5-85d6-9b9266fa58b2" (UID: "5afd39db-ff97-48a5-85d6-9b9266fa58b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:34.719138 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.719135    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b" (OuterVolumeSpecName: "kube-api-access-rkk7b") pod "5afd39db-ff97-48a5-85d6-9b9266fa58b2" (UID: "5afd39db-ff97-48a5-85d6-9b9266fa58b2"). InnerVolumeSpecName "kube-api-access-rkk7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:34.818434 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.818398    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkk7b\" (UniqueName: \"kubernetes.io/projected/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-api-access-rkk7b\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:34.818434 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.818427    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5afd39db-ff97-48a5-85d6-9b9266fa58b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:34.818434 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:34.818436    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5afd39db-ff97-48a5-85d6-9b9266fa58b2-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.006410 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.006388    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:21:35.028154 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.028125    2568 generic.go:356] "Generic (PLEG): container finished" podID="420b74ef-0f55-4f6f-a974-beff045533bb" containerID="eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af" exitCode=0
Jul 31 15:21:35.028595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.028184    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"
Jul 31 15:21:35.028595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.028226    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerDied","Data":"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"}
Jul 31 15:21:35.028595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.028290    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg" event={"ID":"420b74ef-0f55-4f6f-a974-beff045533bb","Type":"ContainerDied","Data":"b506c2ee8c85c6fbed276bc04ac3be5078a67179aaaf939d3a484e4810e77770"}
Jul 31 15:21:35.028595 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.028323    2568 scope.go:117] "RemoveContainer" containerID="9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.030075    2568 generic.go:356] "Generic (PLEG): container finished" podID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerID="0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664" exitCode=0
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.030149    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.030169    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerDied","Data":"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"}
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.030270    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5" event={"ID":"5afd39db-ff97-48a5-85d6-9b9266fa58b2","Type":"ContainerDied","Data":"d12958e70ecee924dd9f358d241588ed24e8c78cd1dfb19d24117679a5715108"}
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.030722    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:21:35.031327 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.031054    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:21:35.064057 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.064030    2568 scope.go:117] "RemoveContainer" containerID="eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"
Jul 31 15:21:35.072371 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.072349    2568 scope.go:117] "RemoveContainer" containerID="9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"
Jul 31 15:21:35.072745 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:35.072709    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48\": container with ID starting with 9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48 not found: ID does not exist" containerID="9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"
Jul 31 15:21:35.072834 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.072754    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48"} err="failed to get container status \"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48\": rpc error: code = NotFound desc = could not find container \"9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48\": container with ID starting with 9228b8b44c3af509fe5875033b187d209361c6e6ed89033f261be13224a05f48 not found: ID does not exist"
Jul 31 15:21:35.072834 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.072774    2568 scope.go:117] "RemoveContainer" containerID="eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"
Jul 31 15:21:35.073123 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:35.073089    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af\": container with ID starting with eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af not found: ID does not exist" containerID="eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"
Jul 31 15:21:35.073302 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.073129    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af"} err="failed to get container status \"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af\": rpc error: code = NotFound desc = could not find container \"eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af\": container with ID starting with eadbc6f704b64cf7ccc37fedc1cf7846115e266bed9fc32eb724b0ebb6e6b4af not found: ID does not exist"
Jul 31 15:21:35.073302 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.073154    2568 scope.go:117] "RemoveContainer" containerID="7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"
Jul 31 15:21:35.077984 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.077022    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:21:35.083596 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.083573    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7b982-predictor-5857f8478-6d7w5"]
Jul 31 15:21:35.090382 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.090346    2568 scope.go:117] "RemoveContainer" containerID="0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"
Jul 31 15:21:35.098722 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.098635    2568 scope.go:117] "RemoveContainer" containerID="7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"
Jul 31 15:21:35.101759 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:35.101656    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679\": container with ID starting with 7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679 not found: ID does not exist" containerID="7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"
Jul 31 15:21:35.101759 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.101693    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679"} err="failed to get container status \"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679\": rpc error: code = NotFound desc = could not find container \"7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679\": container with ID starting with 7daad529951b2c433e422a54cfde20d4defa30401a30e3157a49f1e1ec4ad679 not found: ID does not exist"
Jul 31 15:21:35.101759 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.101723    2568 scope.go:117] "RemoveContainer" containerID="0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"
Jul 31 15:21:35.102695 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:35.102406    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664\": container with ID starting with 0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664 not found: ID does not exist" containerID="0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"
Jul 31 15:21:35.102695 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.102437    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664"} err="failed to get container status \"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664\": rpc error: code = NotFound desc = could not find container \"0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664\": container with ID starting with 0ffe83a6b88718f9d100ec8d97bcca404e3cba4fe284069c30adcafba1b60664 not found: ID does not exist"
Jul 31 15:21:35.120980 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.120957    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lmtz\" (UniqueName: \"kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz\") pod \"420b74ef-0f55-4f6f-a974-beff045533bb\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") "
Jul 31 15:21:35.121108 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.120989    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config\") pod \"420b74ef-0f55-4f6f-a974-beff045533bb\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") "
Jul 31 15:21:35.121108 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.121023    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls\") pod \"420b74ef-0f55-4f6f-a974-beff045533bb\" (UID: \"420b74ef-0f55-4f6f-a974-beff045533bb\") "
Jul 31 15:21:35.121397 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.121372    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "420b74ef-0f55-4f6f-a974-beff045533bb" (UID: "420b74ef-0f55-4f6f-a974-beff045533bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:35.123311 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.123276    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz" (OuterVolumeSpecName: "kube-api-access-8lmtz") pod "420b74ef-0f55-4f6f-a974-beff045533bb" (UID: "420b74ef-0f55-4f6f-a974-beff045533bb"). InnerVolumeSpecName "kube-api-access-8lmtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:35.123390 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.123315    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "420b74ef-0f55-4f6f-a974-beff045533bb" (UID: "420b74ef-0f55-4f6f-a974-beff045533bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:35.183152 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.183131    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:21:35.222027 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.221993    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmtz\" (UniqueName: \"kubernetes.io/projected/420b74ef-0f55-4f6f-a974-beff045533bb-kube-api-access-8lmtz\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.222027 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.222020    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/420b74ef-0f55-4f6f-a974-beff045533bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.222027 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.222029    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/420b74ef-0f55-4f6f-a974-beff045533bb-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.323082 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.323050    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config\") pod \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") "
Jul 31 15:21:35.323252 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.323092    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls\") pod \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") "
Jul 31 15:21:35.323252 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.323140    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hksvc\" (UniqueName: \"kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc\") pod \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\" (UID: \"d90a334f-7eea-41c6-aeaa-a44c41f481f3\") "
Jul 31 15:21:35.323457 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.323433    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d90a334f-7eea-41c6-aeaa-a44c41f481f3" (UID: "d90a334f-7eea-41c6-aeaa-a44c41f481f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:35.325292 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.325273    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d90a334f-7eea-41c6-aeaa-a44c41f481f3" (UID: "d90a334f-7eea-41c6-aeaa-a44c41f481f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:35.325428 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.325405    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc" (OuterVolumeSpecName: "kube-api-access-hksvc") pod "d90a334f-7eea-41c6-aeaa-a44c41f481f3" (UID: "d90a334f-7eea-41c6-aeaa-a44c41f481f3"). InnerVolumeSpecName "kube-api-access-hksvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:35.350478 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.350454    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:21:35.354593 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.354571    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b982-predictor-d56595d99-q8npg"]
Jul 31 15:21:35.423669 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.423635    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.423669 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.423666    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d90a334f-7eea-41c6-aeaa-a44c41f481f3-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.423669 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.423677    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hksvc\" (UniqueName: \"kubernetes.io/projected/d90a334f-7eea-41c6-aeaa-a44c41f481f3-kube-api-access-hksvc\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.636713 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.636688    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:21:35.726417 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.726374    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config\") pod \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") "
Jul 31 15:21:35.726565 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.726480    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") pod \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") "
Jul 31 15:21:35.726565 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.726524    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmppw\" (UniqueName: \"kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw\") pod \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\" (UID: \"5588be0d-175b-4bc7-a25e-91e4bfb55df4\") "
Jul 31 15:21:35.726796 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.726774    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5588be0d-175b-4bc7-a25e-91e4bfb55df4" (UID: "5588be0d-175b-4bc7-a25e-91e4bfb55df4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:35.728753 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.728731    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw" (OuterVolumeSpecName: "kube-api-access-wmppw") pod "5588be0d-175b-4bc7-a25e-91e4bfb55df4" (UID: "5588be0d-175b-4bc7-a25e-91e4bfb55df4"). InnerVolumeSpecName "kube-api-access-wmppw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:35.728800 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.728747    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5588be0d-175b-4bc7-a25e-91e4bfb55df4" (UID: "5588be0d-175b-4bc7-a25e-91e4bfb55df4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:35.827164 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.827067    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.827164 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.827102    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5588be0d-175b-4bc7-a25e-91e4bfb55df4-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:35.827164 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:35.827117    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmppw\" (UniqueName: \"kubernetes.io/projected/5588be0d-175b-4bc7-a25e-91e4bfb55df4-kube-api-access-wmppw\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:21:36.036080 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.036043    2568 generic.go:356] "Generic (PLEG): container finished" podID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerID="b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090" exitCode=0
Jul 31 15:21:36.036555 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.036101    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"
Jul 31 15:21:36.036555 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.036128    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerDied","Data":"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"}
Jul 31 15:21:36.036555 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.036221    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv" event={"ID":"d90a334f-7eea-41c6-aeaa-a44c41f481f3","Type":"ContainerDied","Data":"471b1075d1e45ea5ad1355bc861f904237ec86fd5d174cad47078c2eb9ba0567"}
Jul 31 15:21:36.036555 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.036262    2568 scope.go:117] "RemoveContainer" containerID="78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"
Jul 31 15:21:36.038645 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.038619    2568 generic.go:356] "Generic (PLEG): container finished" podID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerID="4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154" exitCode=0
Jul 31 15:21:36.038756 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.038653    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerDied","Data":"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"}
Jul 31 15:21:36.038756 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.038686    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"
Jul 31 15:21:36.038756 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.038717    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj" event={"ID":"5588be0d-175b-4bc7-a25e-91e4bfb55df4","Type":"ContainerDied","Data":"5017af668f137c99030bd6a3f5f7ce2443526cad218c5f41d68868ff59cb2751"}
Jul 31 15:21:36.039906 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.039878    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:21:36.045734 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.045657    2568 scope.go:117] "RemoveContainer" containerID="b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"
Jul 31 15:21:36.053170 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.053153    2568 scope.go:117] "RemoveContainer" containerID="78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"
Jul 31 15:21:36.053456 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:36.053434    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e\": container with ID starting with 78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e not found: ID does not exist" containerID="78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"
Jul 31 15:21:36.053547 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.053465    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e"} err="failed to get container status \"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e\": rpc error: code = NotFound desc = could not find container \"78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e\": container with ID starting with 78ee823f3472c0938710c9b292b0088964bff6496984d02c1eb91c2e9bc46c6e not found: ID does not exist"
Jul 31 15:21:36.053547 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.053484    2568 scope.go:117] "RemoveContainer" containerID="b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"
Jul 31 15:21:36.053730 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:36.053712    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090\": container with ID starting with b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090 not found: ID does not exist" containerID="b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"
Jul 31 15:21:36.053768 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.053737    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090"} err="failed to get container status \"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090\": rpc error: code = NotFound desc = could not find container \"b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090\": container with ID starting with b6e81663295b1a4eee9d316994b172cce8361a0eae4e2ee20af63894c9d1b090 not found: ID does not exist"
Jul 31 15:21:36.053768 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.053753    2568 scope.go:117] "RemoveContainer" containerID="6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"
Jul 31 15:21:36.059313 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.059291    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:21:36.061406 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.061389    2568 scope.go:117] "RemoveContainer" containerID="4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"
Jul 31 15:21:36.066378 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.066353    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e31d5-predictor-7bf5c4d786-cwrxv"]
Jul 31 15:21:36.069384 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.069369    2568 scope.go:117] "RemoveContainer" containerID="6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"
Jul 31 15:21:36.069674 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:36.069656    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e\": container with ID starting with 6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e not found: ID does not exist" containerID="6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"
Jul 31 15:21:36.069717 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.069682    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e"} err="failed to get container status \"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e\": rpc error: code = NotFound desc = could not find container \"6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e\": container with ID starting with 6ef13a6d6583f97fc7c7e9cba7dfb04dfe3f7793dc6b2debc2cb7ab41e7d038e not found: ID does not exist"
Jul 31 15:21:36.069717 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.069700    2568 scope.go:117] "RemoveContainer" containerID="4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"
Jul 31 15:21:36.069949 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:21:36.069932    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154\": container with ID starting with 4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154 not found: ID does not exist" containerID="4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"
Jul 31 15:21:36.069993 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.069954    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154"} err="failed to get container status \"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154\": rpc error: code = NotFound desc = could not find container \"4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154\": container with ID starting with 4712fbcc8927c5c5bc6a911f8cbde7bd51b7fb0183f4d56ffa37104e0f818154 not found: ID does not exist"
Jul 31 15:21:36.077554 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.077506    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:21:36.082402 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.082375    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e31d5-predictor-8545984775-n6bpj"]
Jul 31 15:21:36.589807 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.589775    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" path="/var/lib/kubelet/pods/420b74ef-0f55-4f6f-a974-beff045533bb/volumes"
Jul 31 15:21:36.590189 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.590175    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" path="/var/lib/kubelet/pods/5588be0d-175b-4bc7-a25e-91e4bfb55df4/volumes"
Jul 31 15:21:36.590559 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.590547    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" path="/var/lib/kubelet/pods/5afd39db-ff97-48a5-85d6-9b9266fa58b2/volumes"
Jul 31 15:21:36.591221 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:36.591185    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" path="/var/lib/kubelet/pods/d90a334f-7eea-41c6-aeaa-a44c41f481f3/volumes"
Jul 31 15:21:37.014445 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:37.014420    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:21:37.014888 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:37.014858    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:21:39.026266 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:39.026234    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:21:39.026702 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:39.026620    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:21:40.035567 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:40.035535    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:21:40.036034 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:40.036006    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:21:41.044575 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:41.044545    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:21:41.045036 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:41.045011    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:21:47.015778 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:47.015734    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:21:49.026830 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:49.026791    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:21:50.036495 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:50.036457    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:21:51.045073 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:51.045031    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:21:57.015498 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:57.015458    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:21:59.026885 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:21:59.026845    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:22:00.036358 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:00.036318    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:22:01.045061 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:01.045011    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:22:07.015833 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:07.015793    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 31 15:22:09.026873 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:09.026828    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:22:10.036160 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:10.036120    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:22:11.045120 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:11.045080    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:22:17.015950 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:17.015920    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:22:19.027040 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:19.027002    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:22:20.036842 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:20.036805    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:22:21.046408 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:21.046371    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:22:29.027424 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:29.027341    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:22:30.036391 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:22:30.036359    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:23:08.566832 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:23:08.566797    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:23:08.568526 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:23:08.568500    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:28:08.599149 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:28:08.599120    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:28:08.600468 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:28:08.600441    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:30:55.482973 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.482938    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:30:55.485522 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.483286    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container" containerID="cri-o://77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9" gracePeriod=30
Jul 31 15:30:55.485522 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.483285    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kube-rbac-proxy" containerID="cri-o://657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b" gracePeriod=30
Jul 31 15:30:55.533735 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.533700    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:30:55.534060 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.533978    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container" containerID="cri-o://607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5" gracePeriod=30
Jul 31 15:30:55.534060 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:55.534042    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kube-rbac-proxy" containerID="cri-o://7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb" gracePeriod=30
Jul 31 15:30:56.041088 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:56.041045    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 31 15:30:56.106340 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:56.106248    2568 generic.go:356] "Generic (PLEG): container finished" podID="5fa64ced-8e39-4ede-9456-34d853453666" containerID="7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb" exitCode=2
Jul 31 15:30:56.106340 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:56.106316    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerDied","Data":"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"}
Jul 31 15:30:56.107852 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:56.107831    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerID="657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b" exitCode=2
Jul 31 15:30:56.107964 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:56.107862    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerDied","Data":"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"}
Jul 31 15:30:58.833650 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.833624    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:30:58.875731 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.875701    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:30:58.993719 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993688    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config\") pod \"5fa64ced-8e39-4ede-9456-34d853453666\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") "
Jul 31 15:30:58.993897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993735    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") pod \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") "
Jul 31 15:30:58.993897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993763    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config\") pod \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") "
Jul 31 15:30:58.993897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993819    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") pod \"5fa64ced-8e39-4ede-9456-34d853453666\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") "
Jul 31 15:30:58.993897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993841    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvpvh\" (UniqueName: \"kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh\") pod \"5fa64ced-8e39-4ede-9456-34d853453666\" (UID: \"5fa64ced-8e39-4ede-9456-34d853453666\") "
Jul 31 15:30:58.993897 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.993866    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms4kd\" (UniqueName: \"kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd\") pod \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\" (UID: \"5b3aad63-8ee8-4381-bffc-2af39bbccc34\") "
Jul 31 15:30:58.994789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.994124    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fa64ced-8e39-4ede-9456-34d853453666" (UID: "5fa64ced-8e39-4ede-9456-34d853453666"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:30:58.994789 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.994134    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b3aad63-8ee8-4381-bffc-2af39bbccc34" (UID: "5b3aad63-8ee8-4381-bffc-2af39bbccc34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:30:58.996126 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.996102    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh" (OuterVolumeSpecName: "kube-api-access-pvpvh") pod "5fa64ced-8e39-4ede-9456-34d853453666" (UID: "5fa64ced-8e39-4ede-9456-34d853453666"). InnerVolumeSpecName "kube-api-access-pvpvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:30:58.996126 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.996109    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd" (OuterVolumeSpecName: "kube-api-access-ms4kd") pod "5b3aad63-8ee8-4381-bffc-2af39bbccc34" (UID: "5b3aad63-8ee8-4381-bffc-2af39bbccc34"). InnerVolumeSpecName "kube-api-access-ms4kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:30:58.996126 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.996118    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fa64ced-8e39-4ede-9456-34d853453666" (UID: "5fa64ced-8e39-4ede-9456-34d853453666"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:30:58.996314 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:58.996118    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b3aad63-8ee8-4381-bffc-2af39bbccc34" (UID: "5b3aad63-8ee8-4381-bffc-2af39bbccc34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:30:59.094527 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094495    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fa64ced-8e39-4ede-9456-34d853453666-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.094527 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094526    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpvh\" (UniqueName: \"kubernetes.io/projected/5fa64ced-8e39-4ede-9456-34d853453666-kube-api-access-pvpvh\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.094527 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094537    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4kd\" (UniqueName: \"kubernetes.io/projected/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-api-access-ms4kd\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.094816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094546    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fa64ced-8e39-4ede-9456-34d853453666-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.094816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094555    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b3aad63-8ee8-4381-bffc-2af39bbccc34-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.094816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.094563    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b3aad63-8ee8-4381-bffc-2af39bbccc34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:30:59.120034 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.119954    2568 generic.go:356] "Generic (PLEG): container finished" podID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerID="77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9" exitCode=0
Jul 31 15:30:59.120034 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.120005    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"
Jul 31 15:30:59.120287 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.120040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerDied","Data":"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"}
Jul 31 15:30:59.120287 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.120098    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg" event={"ID":"5b3aad63-8ee8-4381-bffc-2af39bbccc34","Type":"ContainerDied","Data":"7a7d2ae1c91415321c0f3c31033bab5a322fc16c520908efa16b604226d1194c"}
Jul 31 15:30:59.120287 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.120127    2568 scope.go:117] "RemoveContainer" containerID="657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"
Jul 31 15:30:59.121663 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.121629    2568 generic.go:356] "Generic (PLEG): container finished" podID="5fa64ced-8e39-4ede-9456-34d853453666" containerID="607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5" exitCode=0
Jul 31 15:30:59.121764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.121664    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerDied","Data":"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"}
Jul 31 15:30:59.121764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.121682    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"
Jul 31 15:30:59.121764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.121711    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp" event={"ID":"5fa64ced-8e39-4ede-9456-34d853453666","Type":"ContainerDied","Data":"e5cb6a34f3b6c80f928cb881da1d4b93a50eb5d3adc364eb7b5904f18e058b82"}
Jul 31 15:30:59.128563 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.128537    2568 scope.go:117] "RemoveContainer" containerID="77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"
Jul 31 15:30:59.136220 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.136180    2568 scope.go:117] "RemoveContainer" containerID="657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"
Jul 31 15:30:59.136459 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:30:59.136439    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b\": container with ID starting with 657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b not found: ID does not exist" containerID="657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"
Jul 31 15:30:59.136502 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.136469    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b"} err="failed to get container status \"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b\": rpc error: code = NotFound desc = could not find container \"657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b\": container with ID starting with 657a9ff495d417ed295be6db8d2f4327a1403e45e2504d2a611734874048905b not found: ID does not exist"
Jul 31 15:30:59.136502 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.136486    2568 scope.go:117] "RemoveContainer" containerID="77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"
Jul 31 15:30:59.136743 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:30:59.136725    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9\": container with ID starting with 77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9 not found: ID does not exist" containerID="77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"
Jul 31 15:30:59.136823 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.136749    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9"} err="failed to get container status \"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9\": rpc error: code = NotFound desc = could not find container \"77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9\": container with ID starting with 77a3ec262e9e02eca207d8f93acd78dc7a0d52e70c8f8bdf4b86ebbb93089bb9 not found: ID does not exist"
Jul 31 15:30:59.136823 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.136766    2568 scope.go:117] "RemoveContainer" containerID="7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"
Jul 31 15:30:59.143991 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.143970    2568 scope.go:117] "RemoveContainer" containerID="607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"
Jul 31 15:30:59.146080 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.146061    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:30:59.151606 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.151588    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-faf8d-predictor-6f9fc8c58f-8vbhp"]
Jul 31 15:30:59.151676 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.151656    2568 scope.go:117] "RemoveContainer" containerID="7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"
Jul 31 15:30:59.151919 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:30:59.151901    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb\": container with ID starting with 7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb not found: ID does not exist" containerID="7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"
Jul 31 15:30:59.151982 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.151925    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb"} err="failed to get container status \"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb\": rpc error: code = NotFound desc = could not find container \"7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb\": container with ID starting with 7aad4d3e915470b269a5b5428ce323cde7524d36a6745c3a8bab57c28d2e41cb not found: ID does not exist"
Jul 31 15:30:59.151982 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.151947    2568 scope.go:117] "RemoveContainer" containerID="607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"
Jul 31 15:30:59.152177 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:30:59.152163    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5\": container with ID starting with 607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5 not found: ID does not exist" containerID="607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"
Jul 31 15:30:59.152231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.152180    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5"} err="failed to get container status \"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5\": rpc error: code = NotFound desc = could not find container \"607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5\": container with ID starting with 607120a30acb4a5e77953063aa8754a198d3b0b19dd5773605ec13de16ca06f5 not found: ID does not exist"
Jul 31 15:30:59.162733 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.162702    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:30:59.167575 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:30:59.167555    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-faf8d-predictor-5896849644-7rtxg"]
Jul 31 15:31:00.589454 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:31:00.589425    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" path="/var/lib/kubelet/pods/5b3aad63-8ee8-4381-bffc-2af39bbccc34/volumes"
Jul 31 15:31:00.589868 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:31:00.589806    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa64ced-8e39-4ede-9456-34d853453666" path="/var/lib/kubelet/pods/5fa64ced-8e39-4ede-9456-34d853453666/volumes"
Jul 31 15:33:08.620007 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:33:08.619919    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:33:08.620919 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:33:08.620900    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:38:08.643456 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:08.643427    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:38:08.645935 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:08.644396    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:38:49.795342 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.795308    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:38:49.795837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.795520    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container" containerID="cri-o://1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52" gracePeriod=30
Jul 31 15:38:49.795837 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.795564    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kube-rbac-proxy" containerID="cri-o://3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8" gracePeriod=30
Jul 31 15:38:49.853047 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.853017    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:38:49.853281 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.853256    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container" containerID="cri-o://997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d" gracePeriod=30
Jul 31 15:38:49.853378 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:49.853307    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kube-rbac-proxy" containerID="cri-o://399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893" gracePeriod=30
Jul 31 15:38:50.794977 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:50.794944    2568 generic.go:356] "Generic (PLEG): container finished" podID="3460efc7-de17-47da-89e9-87bcba812bbe" containerID="3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8" exitCode=2
Jul 31 15:38:50.795182 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:50.795025    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerDied","Data":"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"}
Jul 31 15:38:50.796871 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:50.796843    2568 generic.go:356] "Generic (PLEG): container finished" podID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerID="399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893" exitCode=2
Jul 31 15:38:50.797213 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:50.796893    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerDied","Data":"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"}
Jul 31 15:38:52.010333 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.010284    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 31 15:38:52.941793 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.941763    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:38:52.965801 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.965767    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97q2f\" (UniqueName: \"kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f\") pod \"3460efc7-de17-47da-89e9-87bcba812bbe\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") "
Jul 31 15:38:52.965997 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.965827    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls\") pod \"3460efc7-de17-47da-89e9-87bcba812bbe\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") "
Jul 31 15:38:52.965997 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.965901    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config\") pod \"3460efc7-de17-47da-89e9-87bcba812bbe\" (UID: \"3460efc7-de17-47da-89e9-87bcba812bbe\") "
Jul 31 15:38:52.966295 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.966267    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3460efc7-de17-47da-89e9-87bcba812bbe" (UID: "3460efc7-de17-47da-89e9-87bcba812bbe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:38:52.968284 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.968252    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f" (OuterVolumeSpecName: "kube-api-access-97q2f") pod "3460efc7-de17-47da-89e9-87bcba812bbe" (UID: "3460efc7-de17-47da-89e9-87bcba812bbe"). InnerVolumeSpecName "kube-api-access-97q2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:38:52.968408 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:52.968323    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3460efc7-de17-47da-89e9-87bcba812bbe" (UID: "3460efc7-de17-47da-89e9-87bcba812bbe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:38:53.066426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.066348    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3460efc7-de17-47da-89e9-87bcba812bbe-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.066426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.066382    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3460efc7-de17-47da-89e9-87bcba812bbe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.066426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.066396    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97q2f\" (UniqueName: \"kubernetes.io/projected/3460efc7-de17-47da-89e9-87bcba812bbe-kube-api-access-97q2f\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.188418 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.188393    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:38:53.267739 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.267705    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config\") pod \"1c74c8bc-28a4-467a-a807-0e14cfdad979\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") "
Jul 31 15:38:53.267907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.267751    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzxwj\" (UniqueName: \"kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj\") pod \"1c74c8bc-28a4-467a-a807-0e14cfdad979\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") "
Jul 31 15:38:53.267907 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.267774    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls\") pod \"1c74c8bc-28a4-467a-a807-0e14cfdad979\" (UID: \"1c74c8bc-28a4-467a-a807-0e14cfdad979\") "
Jul 31 15:38:53.268159 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.268126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c74c8bc-28a4-467a-a807-0e14cfdad979" (UID: "1c74c8bc-28a4-467a-a807-0e14cfdad979"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:38:53.270140 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.270112    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c74c8bc-28a4-467a-a807-0e14cfdad979" (UID: "1c74c8bc-28a4-467a-a807-0e14cfdad979"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:38:53.270140 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.270128    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj" (OuterVolumeSpecName: "kube-api-access-zzxwj") pod "1c74c8bc-28a4-467a-a807-0e14cfdad979" (UID: "1c74c8bc-28a4-467a-a807-0e14cfdad979"). InnerVolumeSpecName "kube-api-access-zzxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:38:53.368319 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.368236    2568 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.368319 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.368266    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxwj\" (UniqueName: \"kubernetes.io/projected/1c74c8bc-28a4-467a-a807-0e14cfdad979-kube-api-access-zzxwj\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.368319 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.368276    2568 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c74c8bc-28a4-467a-a807-0e14cfdad979-proxy-tls\") on node \"ip-10-0-135-216.ec2.internal\" DevicePath \"\""
Jul 31 15:38:53.810573 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.810545    2568 generic.go:356] "Generic (PLEG): container finished" podID="3460efc7-de17-47da-89e9-87bcba812bbe" containerID="1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52" exitCode=0
Jul 31 15:38:53.810735 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.810595    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"
Jul 31 15:38:53.810735 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.810635    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerDied","Data":"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"}
Jul 31 15:38:53.810735 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.810696    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg" event={"ID":"3460efc7-de17-47da-89e9-87bcba812bbe","Type":"ContainerDied","Data":"771b70b51b5039d11df66f1e69043a2f5f996b69f3a1cd96d98a5eb6b4c989d6"}
Jul 31 15:38:53.810875 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.810737    2568 scope.go:117] "RemoveContainer" containerID="3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"
Jul 31 15:38:53.812332 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.812305    2568 generic.go:356] "Generic (PLEG): container finished" podID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerID="997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d" exitCode=0
Jul 31 15:38:53.812578 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.812354    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerDied","Data":"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"}
Jul 31 15:38:53.812578 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.812405    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv" event={"ID":"1c74c8bc-28a4-467a-a807-0e14cfdad979","Type":"ContainerDied","Data":"65c7a4736814015736aafa5e14d47b09adfce0f113918c4808fc753806ecfd14"}
Jul 31 15:38:53.812578 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.812357    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"
Jul 31 15:38:53.820077 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.820001    2568 scope.go:117] "RemoveContainer" containerID="1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"
Jul 31 15:38:53.828033 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.828014    2568 scope.go:117] "RemoveContainer" containerID="3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"
Jul 31 15:38:53.828316 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:38:53.828296    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8\": container with ID starting with 3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8 not found: ID does not exist" containerID="3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"
Jul 31 15:38:53.828425 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.828330    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8"} err="failed to get container status \"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8\": rpc error: code = NotFound desc = could not find container \"3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8\": container with ID starting with 3a1e6f918ee06bdbadd841026d289ebaaab3072c62c6d563017bb484050bdcd8 not found: ID does not exist"
Jul 31 15:38:53.828425 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.828352    2568 scope.go:117] "RemoveContainer" containerID="1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"
Jul 31 15:38:53.828622 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:38:53.828606    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52\": container with ID starting with 1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52 not found: ID does not exist" containerID="1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"
Jul 31 15:38:53.828667 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.828629    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52"} err="failed to get container status \"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52\": rpc error: code = NotFound desc = could not find container \"1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52\": container with ID starting with 1c2220fdd50dbeff7f6f8a1b04e664081d4aac5a9b1f080ab39d9baf99220c52 not found: ID does not exist"
Jul 31 15:38:53.828667 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.828644    2568 scope.go:117] "RemoveContainer" containerID="399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"
Jul 31 15:38:53.836062 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.835926    2568 scope.go:117] "RemoveContainer" containerID="997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"
Jul 31 15:38:53.836769 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.836745    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:38:53.841721 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.841700    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7902a-predictor-8dd58f79b-bjhqg"]
Jul 31 15:38:53.843875 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.843859    2568 scope.go:117] "RemoveContainer" containerID="399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"
Jul 31 15:38:53.844179 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:38:53.844162    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893\": container with ID starting with 399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893 not found: ID does not exist" containerID="399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"
Jul 31 15:38:53.844278 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.844187    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893"} err="failed to get container status \"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893\": rpc error: code = NotFound desc = could not find container \"399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893\": container with ID starting with 399af11c6c36866048b15ee64fc76c0eee941f50d5c6077a74ff3a415f792893 not found: ID does not exist"
Jul 31 15:38:53.844278 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.844231    2568 scope.go:117] "RemoveContainer" containerID="997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"
Jul 31 15:38:53.844467 ip-10-0-135-216 kubenswrapper[2568]: E0731 15:38:53.844448    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d\": container with ID starting with 997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d not found: ID does not exist" containerID="997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"
Jul 31 15:38:53.844509 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.844474    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d"} err="failed to get container status \"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d\": rpc error: code = NotFound desc = could not find container \"997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d\": container with ID starting with 997a00e7df1b39c3e76180b9e92b86f7ca7578361347f701ad41fb69517aa41d not found: ID does not exist"
Jul 31 15:38:53.853286 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.853259    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:38:53.857489 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:53.857469    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7902a-predictor-c5f95cb9c-sqvtv"]
Jul 31 15:38:54.588865 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:54.588828    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" path="/var/lib/kubelet/pods/1c74c8bc-28a4-467a-a807-0e14cfdad979/volumes"
Jul 31 15:38:54.589320 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:38:54.589305    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" path="/var/lib/kubelet/pods/3460efc7-de17-47da-89e9-87bcba812bbe/volumes"
Jul 31 15:39:25.661272 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:25.661230    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kbpzc_f0ab4de6-e675-43b2-9930-5c43ad601b61/global-pull-secret-syncer/0.log"
Jul 31 15:39:25.865007 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:25.864980    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wzpsj_e3d4e8c6-8adc-4875-b2cf-4e4209776626/konnectivity-agent/0.log"
Jul 31 15:39:25.894426 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:25.894392    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-216.ec2.internal_bd551728a37b2cd396a15397d027448d/haproxy/0.log"
Jul 31 15:39:30.677663 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:30.677629    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pmzv_8a42b7dd-571c-43d1-b36d-7d0d8b20540d/node-exporter/0.log"
Jul 31 15:39:30.703524 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:30.703494    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pmzv_8a42b7dd-571c-43d1-b36d-7d0d8b20540d/kube-rbac-proxy/0.log"
Jul 31 15:39:30.731396 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:30.731368    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2pmzv_8a42b7dd-571c-43d1-b36d-7d0d8b20540d/init-textfile/0.log"
Jul 31 15:39:33.722777 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.722745    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"]
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723018    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723030    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723038    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723043    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723051    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723057    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723065    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723070    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723080    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723090    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723101    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723106    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723113    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723118    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723127    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723131    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723138    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723143    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723151    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723156    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723167    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723176    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723184    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723189    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723208    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723213    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723219    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kube-rbac-proxy"
Jul 31 15:39:33.723214 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723224    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723231    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723236    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723242    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723250    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723298    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723307    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90a334f-7eea-41c6-aeaa-a44c41f481f3" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723313    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723318    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723323    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723329    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723339    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723346    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723353    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3aad63-8ee8-4381-bffc-2af39bbccc34" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723357    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="420b74ef-0f55-4f6f-a974-beff045533bb" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723364    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723368    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c74c8bc-28a4-467a-a807-0e14cfdad979" containerName="kserve-container"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723373    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa64ced-8e39-4ede-9456-34d853453666" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723378    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3460efc7-de17-47da-89e9-87bcba812bbe" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723384    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5588be0d-175b-4bc7-a25e-91e4bfb55df4" containerName="kube-rbac-proxy"
Jul 31 15:39:33.724231 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.723389    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5afd39db-ff97-48a5-85d6-9b9266fa58b2" containerName="kube-rbac-proxy"
Jul 31 15:39:33.726111 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.726094    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.729334 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.729314    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd56f\"/\"openshift-service-ca.crt\""
Jul 31 15:39:33.729454 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.729314    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cd56f\"/\"default-dockercfg-mzp76\""
Jul 31 15:39:33.731434 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.731402    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cd56f\"/\"kube-root-ca.crt\""
Jul 31 15:39:33.738037 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.738011    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"]
Jul 31 15:39:33.766614 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.766578    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-podres\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.766764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.766628    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-sys\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.766764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.766654    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-lib-modules\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.766764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.766711    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ngx\" (UniqueName: \"kubernetes.io/projected/f1447639-2f26-4a40-ac05-5a09add11ee7-kube-api-access-q4ngx\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.766764 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.766744    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-proc\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.867851 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.867810    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ngx\" (UniqueName: \"kubernetes.io/projected/f1447639-2f26-4a40-ac05-5a09add11ee7-kube-api-access-q4ngx\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.867851 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.867854    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-proc\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868048 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.867911    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-podres\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868048 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.867947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-sys\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868048 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.867975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-lib-modules\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868048 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.868020    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-proc\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868048 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.868032    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-sys\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868259 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.868075    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-podres\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.868259 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.868104    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1447639-2f26-4a40-ac05-5a09add11ee7-lib-modules\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:33.877914 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:33.877895    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ngx\" (UniqueName: \"kubernetes.io/projected/f1447639-2f26-4a40-ac05-5a09add11ee7-kube-api-access-q4ngx\") pod \"perf-node-gather-daemonset-p99hl\" (UID: \"f1447639-2f26-4a40-ac05-5a09add11ee7\") " pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:34.035965 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.035864    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:34.173687 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.173653    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"]
Jul 31 15:39:34.176640 ip-10-0-135-216 kubenswrapper[2568]: W0731 15:39:34.176608    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1447639_2f26_4a40_ac05_5a09add11ee7.slice/crio-478ddc1268b640ad88a412bea5832fdcb6fb691d97278e1db8e5b1d28846d3c6 WatchSource:0}: Error finding container 478ddc1268b640ad88a412bea5832fdcb6fb691d97278e1db8e5b1d28846d3c6: Status 404 returned error can't find the container with id 478ddc1268b640ad88a412bea5832fdcb6fb691d97278e1db8e5b1d28846d3c6
Jul 31 15:39:34.961579 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.961534    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl" event={"ID":"f1447639-2f26-4a40-ac05-5a09add11ee7","Type":"ContainerStarted","Data":"486d3be02a0cf824204d10ebf777e775df7323d64eeaf13698dd78a8fb5a67a5"}
Jul 31 15:39:34.961995 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.961593    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl" event={"ID":"f1447639-2f26-4a40-ac05-5a09add11ee7","Type":"ContainerStarted","Data":"478ddc1268b640ad88a412bea5832fdcb6fb691d97278e1db8e5b1d28846d3c6"}
Jul 31 15:39:34.961995 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.961631    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:34.979463 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:34.979409    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl" podStartSLOduration=1.9793917890000001 podStartE2EDuration="1.979391789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:39:34.978162303 +0000 UTC m=+3086.932476208" watchObservedRunningTime="2026-07-31 15:39:34.979391789 +0000 UTC m=+3086.933705694"
Jul 31 15:39:35.536274 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:35.536249    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cn8d_bddbb76d-dfbc-41aa-b6ca-41389433dafd/dns/0.log"
Jul 31 15:39:35.596999 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:35.596962    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9cn8d_bddbb76d-dfbc-41aa-b6ca-41389433dafd/kube-rbac-proxy/0.log"
Jul 31 15:39:35.740307 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:35.740283    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mh4z7_e130e647-10ab-4777-a1ff-a3c083c60005/dns-node-resolver/0.log"
Jul 31 15:39:36.298368 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:36.298332    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rc6lj_d003d921-8080-41f5-a0fa-65086d2e7f14/node-ca/0.log"
Jul 31 15:39:37.580938 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:37.580908    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9httq_bb6c2d69-bf91-4966-b851-e84b6f31f4c7/serve-healthcheck-canary/0.log"
Jul 31 15:39:38.140253 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:38.140221    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bglfl_debb2a9b-0bae-4fef-9de3-8620374ed8f1/kube-rbac-proxy/0.log"
Jul 31 15:39:38.164681 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:38.164655    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bglfl_debb2a9b-0bae-4fef-9de3-8620374ed8f1/exporter/0.log"
Jul 31 15:39:38.187786 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:38.187756    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bglfl_debb2a9b-0bae-4fef-9de3-8620374ed8f1/extractor/0.log"
Jul 31 15:39:40.837898 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:40.837873    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-7vvtz_62836109-e2e3-4959-8013-c742aa942aea/server/0.log"
Jul 31 15:39:40.976045 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:40.976017    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cd56f/perf-node-gather-daemonset-p99hl"
Jul 31 15:39:41.000903 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:41.000874    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-ngp59_cdc048e3-f3f1-459a-b284-396f75448387/manager/0.log"
Jul 31 15:39:41.167695 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:41.167601    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-q5kv8_1f7bf717-b357-4723-bf47-a5a4b84a42ac/seaweedfs/0.log"
Jul 31 15:39:47.551534 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.551503    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:39:47.587740 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.587716    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/egress-router-binary-copy/0.log"
Jul 31 15:39:47.630302 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.630274    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/cni-plugins/0.log"
Jul 31 15:39:47.658816 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.658768    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/bond-cni-plugin/0.log"
Jul 31 15:39:47.682804 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.682764    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/routeoverride-cni/0.log"
Jul 31 15:39:47.708001 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.707974    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/whereabouts-cni-bincopy/0.log"
Jul 31 15:39:47.733970 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:47.733943    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ltrl7_0888bf70-75f8-4ecb-8d07-db4ddceb29c2/whereabouts-cni/0.log"
Jul 31 15:39:48.031005 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:48.030974    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zp5fn_7bdf2867-f4e8-43a2-b790-61b9b5591436/kube-multus/0.log"
Jul 31 15:39:48.106135 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:48.106095    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lkpvs_9680aba1-e848-493d-929a-02fa6fc70f42/network-metrics-daemon/0.log"
Jul 31 15:39:48.127305 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:48.127278    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lkpvs_9680aba1-e848-493d-929a-02fa6fc70f42/kube-rbac-proxy/0.log"
Jul 31 15:39:49.159376 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.159337    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-controller/0.log"
Jul 31 15:39:49.178915 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.178887    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/0.log"
Jul 31 15:39:49.179102 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.178986    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovn-acl-logging/1.log"
Jul 31 15:39:49.205094 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.205062    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/kube-rbac-proxy-node/0.log"
Jul 31 15:39:49.237791 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.237761    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/kube-rbac-proxy-ovn-metrics/0.log"
Jul 31 15:39:49.259868 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.259837    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/northd/0.log"
Jul 31 15:39:49.280093 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.280058    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/nbdb/0.log"
Jul 31 15:39:49.302031 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.302002    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/sbdb/0.log"
Jul 31 15:39:49.359343 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:49.359312    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b95kt_d52edadf-6b08-4eda-8a43-c5d889d953fe/ovnkube-controller/0.log"
Jul 31 15:39:50.669705 ip-10-0-135-216 kubenswrapper[2568]: I0731 15:39:50.669671    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zqnxm_2961b89c-a482-4eb0-832a-b8b31443d0ed/network-check-target-container/0.log"