Jul 31 14:46:06.135621 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 14:46:06.135633 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 14:46:06.135640 ip-10-0-143-227 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 14:46:06.136001 ip-10-0-143-227 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 14:46:16.222633 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 14:46:16.222654 ip-10-0-143-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4df54022964c41818d663eafb6ec3536 --
Jul 31 14:47:56.538929 ip-10-0-143-227 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:47:56.951301 ip-10-0-143-227 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:56.951301 ip-10-0-143-227 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:47:56.951301 ip-10-0-143-227 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:56.951301 ip-10-0-143-227 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:47:56.951301 ip-10-0-143-227 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:56.953849 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.953754    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:47:56.957579 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957557    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.957579 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957581    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957586    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957590    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957594    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957611    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957614    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957617    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957621    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957624    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957627    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957629    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957632    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957635    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957637    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957645    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957648    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957651    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957653    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957656    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.957665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957659    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957661    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957664    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957666    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957669    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957672    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957675    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957677    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957681    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957683    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957686    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957688    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957691    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957693    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957696    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957698    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957701    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957703    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957708    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.958120 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957711    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957714    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957717    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957720    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957723    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957726    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957729    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957732    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957735    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957738    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957741    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957743    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957746    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957748    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957751    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957754    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957757    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957759    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957762    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957765    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.958576 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957767    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957769    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957772    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957775    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957777    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957779    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957782    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957784    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957787    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957790    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957793    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957796    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957800    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957803    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957806    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957809    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957811    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957814    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957817    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.959079 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957820    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957822    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957825    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957828    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957830    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957833    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957835    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957838    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957841    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957843    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957846    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957850    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957853    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957855    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957858    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957860    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957863    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957866    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957868    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957871    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.957873    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.959563 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958616    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958623    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958626    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958629    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958632    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958635    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958638    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958641    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958644    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958647    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958650    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958653    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958656    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958659    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958661    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958665    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958668    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958670    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958673    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958676    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.960129 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958679    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958682    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958684    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958687    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958690    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958692    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958695    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958698    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958700    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958703    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958705    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958710    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958714    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958717    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958720    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958723    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958725    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958728    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958730    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958733    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.960650 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958736    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958739    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958742    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958745    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958747    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958750    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958753    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958756    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958759    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958761    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958764    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958767    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958770    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958773    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958778    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958782    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958784    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958787    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958790    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.961144 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958793    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958795    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958799    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958801    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958803    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958807    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958809    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958812    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958815    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958817    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958820    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958822    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958825    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958827    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958830    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958833    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958837    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958840    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958843    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.961621 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958846    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958849    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958851    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958854    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958857    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958860    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958862    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958865    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958867    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958870    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958873    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958875    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958878    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958880    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958883    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958885    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958888    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958890    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958894    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958897    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.962082 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.958899    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959393    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959400    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959404    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959407    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959410    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959413    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959416    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959419    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959421    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959424    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959426    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959429    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959432    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959435    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959438    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959441    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959444    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959446    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959448    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.962558 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959451    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959454    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959457    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959459    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959463    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959466    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959470    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959473    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959476    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959478    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959482    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959485    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959488    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959490    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959493    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959497    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959501    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959504    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959507    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.963064 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959509    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959512    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959515    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959517    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959520    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959522    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959525    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959527    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959530    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959533    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959536    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959538    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959540    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959543    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959546    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959548    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959551    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959553    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959555    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.963523 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959558    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959560    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959565    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959567    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959571    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959574    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959577    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959580    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959583    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959585    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959588    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959591    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959593    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959610    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959613    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959615    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959618    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959621    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959623    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959626    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.964022 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959629    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959632    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959634    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959637    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959640    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959643    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959646    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959648    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959653    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959657    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959660    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959663    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959665    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959668    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959671    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959673    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959677    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959680    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959683    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.964522 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959686    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959689    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.959692    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959772    2575 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959792    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959801    2575 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959809    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959814    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959817    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959822    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959827    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959830    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959833    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959837    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959841    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959844    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959847    2575 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959851    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959854    2575 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959861    2575 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959864    2575 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959871    2575 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959874    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:47:56.965035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959877    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959880    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959883    2575 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959887    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959890    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959894    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959897    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959901    2575 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959904    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959907    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959911    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959916    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959923    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959927    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959930    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959932    2575 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959936    2575 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959939    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959943    2575 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959946    2575 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959949    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959953    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959956    2575 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959960    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:47:56.965637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959963    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959966    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959969    2575 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959980    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959984    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959987    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959992    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959995    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.959998    2575 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960001    2575 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960010    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960013    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960016    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960019    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960023    2575 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960026    2575 flags.go:64] FLAG: --help="false"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960029    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-227.ec2.internal"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960032    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960035    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960038    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960042    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960045    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960048    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:47:56.966261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960051    2575 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960055    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960057    2575 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960060    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960064    2575 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960066    2575 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960069    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960072    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960075    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960078    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960081    2575 flags.go:64] FLAG: --lock-file=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960084    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960087    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960092    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960098    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960102    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960105    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960108    2575 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960111    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960115    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960118    2575 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960121    2575 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960125    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960129    2575 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960133    2575 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:47:56.966872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960136    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960139    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960142    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960146    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960149    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960152    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960155    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960164    2575 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960167    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960170    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960174    2575 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960177    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960183    2575 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960186    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960190    2575 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960193    2575 flags.go:64] FLAG: --port="10250"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960196    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960199    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d93cf55ced3b8585"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960202    2575 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960205    2575 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960209    2575 flags.go:64] FLAG: --register-node="true"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960213    2575 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960220    2575 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960223    2575 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:47:56.967620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960226    2575 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960229    2575 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960233    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960236    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960240    2575 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960242    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960245    2575 flags.go:64] FLAG: --runonce="false"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960248    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960252    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960255    2575 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960257    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960261    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960264    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960267    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960270    2575 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960274    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960277    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960279    2575 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960282    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960285    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960288    2575 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960291    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960296    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960299    2575 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960302    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:47:56.968456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960307    2575 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960310    2575 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960312    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960315    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960319    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960324    2575 flags.go:64] FLAG: --v="2"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960328    2575 flags.go:64] FLAG: --version="false"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960332    2575 flags.go:64] FLAG: --vmodule=""
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960336    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960340    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960460    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960463    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960466    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960469    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960472    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960474    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960477    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960480    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960483    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960485    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960488    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960491    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.969133 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960493    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960497    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960501    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960505    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960508    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960511    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960514    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960517    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960520    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960523    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960527    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960531    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960533    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960537    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960541    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960545    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960549    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960551    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.969702 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960554    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960557    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960559    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960562    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960564    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960567    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960570    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960572    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960575    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960578    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960580    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960583    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960585    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960587    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960590    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960593    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960610    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960613    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960616    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960619    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.970142 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960621    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960624    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960626    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960629    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960632    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960635    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960638    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960641    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960645    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960649    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960652    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960655    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960658    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960661    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960663    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960666    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960669    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960671    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960674    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960677    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.970665 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960679    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960682    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960684    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960687    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960690    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960692    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960695    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960697    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960700    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960702    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960705    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960707    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960710    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960712    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960715    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960717    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960720    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960722    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960725    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960728    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.971164 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960732    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960735    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960738    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960741    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960744    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960747    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960750    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960752    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.960755    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.960760    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969085    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969106    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969128    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969138    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969222    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.971699 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969228    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969232    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969235    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969239    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969242    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969244    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969247    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969250    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969253    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969255    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969258    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969261    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969264    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969266    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969270    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969272    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969275    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969278    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969281    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.972096 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969284    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969287    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969289    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969293    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969297    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969300    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969303    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969306    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969309    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969313    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969316    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969318    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969321    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969324    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969326    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969329    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969332    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969334    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969337    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969339    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.972561 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969342    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969345    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969348    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969351    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969353    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969357    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969362    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969365    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969367    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969370    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969373    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969375    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969378    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969380    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969383    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969385    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969388    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969391    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969394    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969396    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.973095 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969399    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969402    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969405    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969408    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969410    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969413    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969416    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969418    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969421    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969423    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969426    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969428    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969431    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969433    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969436    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969439    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969441    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969444    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969446    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969449    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.973594 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969452    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969454    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969457    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969459    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969462    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969465    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969467    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969470    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969472    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969474    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969477    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969480    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969483    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969486    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969489    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969493    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969497    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969500    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.974139 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969503    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969508    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969642    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969648    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969651    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969654    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969657    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969660    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969663    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969666    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969669    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969672    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969675    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969677    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969680    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:56.974584 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969683    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969686    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969688    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969691    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969694    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969696    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969699    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969702    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969704    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969707    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969710    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969712    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969716    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969719    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969721    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969724    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969727    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969730    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969732    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:56.974974 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969735    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969737    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969740    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969743    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969745    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969748    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969750    2575 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969753    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969755    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969758    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969760    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969763    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969765    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969768    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969771    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969774    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969777    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969780    2575 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969783    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:56.975458 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969786    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969790    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969793    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969795    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969798    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969801    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969804    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969806    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969809    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969811    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969814    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969817    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969820    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969823    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969825    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969828    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969830    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969833    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969835    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969838    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:56.975938 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969840    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969843    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969846    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969849    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969852    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969854    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969857    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969860    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969863    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969866    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969869    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969872    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969874    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969877    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969879    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969882    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969884    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969887    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969890    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969893    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:56.976432 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969896    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969899    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969902    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969904    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969908    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969911    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969914    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:56.969917    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.969922    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.970061    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.972171    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.973514    2575 server.go:1019] "Starting client certificate rotation"
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.973632    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:47:56.976960 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.973670    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:47:56.977297 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.976560    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:47:56.979644 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.979620    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:47:56.995581 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.995550    2575 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:47:56.997748 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.997729    2575 log.go:25] "Validated CRI v1 image API"
Jul 31 14:47:56.999486 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:56.999470    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:47:57.002733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.002706    2575 fs.go:135] Filesystem UUIDs: map[4b131a73-ea51-480b-b466-8d3ea2286f47:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a9b9ddc7-0e27-48cd-af6a-f74b909586a1:/dev/nvme0n1p3]
Jul 31 14:47:57.002805 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.002731    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:47:57.004717 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.004699    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:47:57.010914 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.010786    2575 manager.go:216] Machine: {Timestamp:2026-07-31 14:47:57.00887876 +0000 UTC m=+0.342998673 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095195 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22fcf32bcfa0a5c0e3a703f10ac7de SystemUUID:ec22fcf3-2bcf-a0a5-c0e3-a703f10ac7de BootID:4df54022-964c-4181-8d66-3eafb6ec3536 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e7:22:55:ca:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e7:22:55:ca:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:7e:cb:f6:f1:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:47:57.010914 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.010902    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:47:57.011070 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.011049    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:47:57.014883 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.014844    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:47:57.015034 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.014888    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:47:57.016207 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.016192    2575 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:47:57.016246 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.016210    2575 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:47:57.016246 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.016219    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:47:57.016906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.016894    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:47:57.018169 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.018156    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:47:57.021328 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021313    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:47:57.021469 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021459    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:47:57.021501 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021493    2575 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:47:57.021532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021503    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:47:57.021532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021523    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:47:57.021583 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021534    2575 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:47:57.021583 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.021543    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:47:57.022032 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.022019    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:47:57.022090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.022038    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:47:57.024320 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.024298    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:47:57.025707 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.025692    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:47:57.025760 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.025723    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:47:57.026368 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026354    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:47:57.026402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026376    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:47:57.026402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026382    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:47:57.026402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026388    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:47:57.026402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026396    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:47:57.026402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026401    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:47:57.026742 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026732    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:47:57.026742 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026742    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:47:57.026796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026751    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:47:57.026796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026757    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:47:57.026796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026788    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:47:57.026796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.026797    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:47:57.027695 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.027683    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:47:57.027695 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.027695    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:47:57.029689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.029669    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hsscq"
Jul 31 14:47:57.031246 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.031232    2575 server.go:1268] "Started kubelet"
Jul 31 14:47:57.031403 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.031338    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:47:57.031452 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.031439    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:47:57.031525 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.031505    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:47:57.032410 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.032299    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:47:57.032410 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.032316    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:47:57.032329 ip-10-0-143-227 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:47:57.033181 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.032471    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:47:57.033181 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.032548    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:47:57.034959 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.034944    2575 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:47:57.037039 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.037017    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:47:57.038019 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.038002    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hsscq"
Jul 31 14:47:57.038891 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.037891    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-227.ec2.internal.18c7673a389bfe18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-227.ec2.internal,UID:ip-10-0-143-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-227.ec2.internal,},FirstTimestamp:2026-07-31 14:47:57.031202328 +0000 UTC m=+0.365322241,LastTimestamp:2026-07-31 14:47:57.031202328 +0000 UTC m=+0.365322241,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-227.ec2.internal,}"
Jul 31 14:47:57.040706 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.040690    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:47:57.041244 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.041223    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:47:57.041473 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.041452    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.041878 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.041862    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:47:57.042104 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042085    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:47:57.042104 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042104    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:47:57.042260 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042168    2575 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:47:57.042260 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042178    2575 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:47:57.042751 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042731    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:47:57.042751 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042751    2575 factory.go:55] Registering systemd factory
Jul 31 14:47:57.042880 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.042761    2575 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:47:57.043025 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.043012    2575 factory.go:153] Registering CRI-O factory
Jul 31 14:47:57.043025 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.043026    2575 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:47:57.043122 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.043048    2575 factory.go:103] Registering Raw factory
Jul 31 14:47:57.043122 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.043061    2575 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:47:57.043501 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.043484    2575 manager.go:349] Starting recovery of all containers
Jul 31 14:47:57.045239 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.045205    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:47:57.045331 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.045290    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:47:57.055246 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.055204    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:47:57.055906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.055884    2575 manager.go:354] Recovery completed
Jul 31 14:47:57.060826 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.060811    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.062578 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.062558    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.062684 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.062590    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.062684 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.062646    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.063223 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063210    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.063281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063210    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:47:57.063281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063240    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:47:57.063281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063256    2575 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:47:57.063917 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063896    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.064013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063930    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.064013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.063944    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.064421 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.064409    2575 policy_none.go:49] "None policy: Start"
Jul 31 14:47:57.064458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.064427    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:47:57.064458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.064437    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:47:57.065426 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.065415    2575 policy_none.go:47] "Start"
Jul 31 14:47:57.107790 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.107630    2575 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:47:57.107927 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.107806    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:47:57.107927 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.107817    2575 server.go:87] "Starting device plugin registration server"
Jul 31 14:47:57.108050 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.108040    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:47:57.108086 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.108052    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:47:57.108185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.108157    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:47:57.108270 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.108260    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:47:57.108270 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.108269    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:47:57.108877 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.108859    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:47:57.108974 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.108902    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.195705 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.195679    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:47:57.195866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.195726    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:47:57.195866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.195752    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:47:57.195866 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.195790    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:47:57.198274 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.198251    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:57.208351 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.208293    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.209277 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.209258    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.209348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.209294    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.209348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.209304    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.209348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.209329    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.220322 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.220301    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.220415 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.220331    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-227.ec2.internal\": node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.238279 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.238258    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.296269 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.296219    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"]
Jul 31 14:47:57.296357 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.296338    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.297955 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.297939    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.298017 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.297969    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.298017 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.297979    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.299380 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.299365    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.299485 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.299472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.299523 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.299501    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.300173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300155    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.300259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300190    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.300259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300204    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.300259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300156    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.300259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300257    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.300486 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.300268    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.301538 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.301521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.301633 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.301554    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:57.302232 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.302217    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:57.302300 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.302244    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:57.302300 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.302258    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:57.329460 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.329428    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-227.ec2.internal\" not found" node="ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.334079 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.334059    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-227.ec2.internal\" not found" node="ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.338738 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.338716    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.344121 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.344093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.344251 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.344133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.344251 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.344161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa64f70662f34d6016d67055768843e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"caa64f70662f34d6016d67055768843e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.439245 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.439209    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.444697 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.444830 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.444830 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa64f70662f34d6016d67055768843e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"caa64f70662f34d6016d67055768843e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.444830 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa64f70662f34d6016d67055768843e-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"caa64f70662f34d6016d67055768843e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.444830 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444783    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.444962 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.444820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0396405a5f8273b532964273d7d51214-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"0396405a5f8273b532964273d7d51214\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.540223 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.540149    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.631722 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.631678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.636688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.636392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:57.640637 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.640620    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.741162 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.741120    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.841704 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.841594    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.907332 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.907300    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:57.942378 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:57.942337    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:57.972687 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.972653    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:47:57.973173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.972844    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:47:57.973173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:57.972846    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:47:58.043234 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:58.043198    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:58.044610 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.044560    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:42:57 +0000 UTC" deadline="2027-12-30 15:59:05.064084625 +0000 UTC"
Jul 31 14:47:58.044672 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.044615    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12409h11m7.019474133s"
Jul 31 14:47:58.131898 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:58.131854    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0396405a5f8273b532964273d7d51214.slice/crio-5f8f02d0c1cc16c947097890aa1aaaf9d5bc4a2766df88f28c03262acd0e9d6f WatchSource:0}: Error finding container 5f8f02d0c1cc16c947097890aa1aaaf9d5bc4a2766df88f28c03262acd0e9d6f: Status 404 returned error can't find the container with id 5f8f02d0c1cc16c947097890aa1aaaf9d5bc4a2766df88f28c03262acd0e9d6f
Jul 31 14:47:58.132494 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:47:58.132475    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa64f70662f34d6016d67055768843e.slice/crio-722aa3b917547f5d4a7541751cd4260191e3d0e1bcfc249f318dc10ff8262961 WatchSource:0}: Error finding container 722aa3b917547f5d4a7541751cd4260191e3d0e1bcfc249f318dc10ff8262961: Status 404 returned error can't find the container with id 722aa3b917547f5d4a7541751cd4260191e3d0e1bcfc249f318dc10ff8262961
Jul 31 14:47:58.139866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.139842    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:47:58.143575 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:58.143552    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 31 14:47:58.169449 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.169422    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:58.198894 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.198840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" event={"ID":"caa64f70662f34d6016d67055768843e","Type":"ContainerStarted","Data":"722aa3b917547f5d4a7541751cd4260191e3d0e1bcfc249f318dc10ff8262961"}
Jul 31 14:47:58.199891 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.199867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"0396405a5f8273b532964273d7d51214","Type":"ContainerStarted","Data":"5f8f02d0c1cc16c947097890aa1aaaf9d5bc4a2766df88f28c03262acd0e9d6f"}
Jul 31 14:47:58.242211 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.242182    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:58.242314 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.242293    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:58.254943 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.254913    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:47:58.256017 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.256001    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 31 14:47:58.264359 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.264339    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:47:58.864992 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:58.864957    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:59.022357 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.022320    2575 apiserver.go:52] "Watching apiserver"
Jul 31 14:47:59.032088 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.032057    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:47:59.034982 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.034933    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nt9j7","openshift-cluster-node-tuning-operator/tuned-lmgcm","openshift-dns/node-resolver-94z8k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal","openshift-multus/multus-h72g8","openshift-multus/network-metrics-daemon-xcfwb","openshift-network-diagnostics/network-check-target-dt4ln","openshift-network-operator/iptables-alerter-5psbp","kube-system/konnectivity-agent-rgwjh","kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck","openshift-image-registry/node-ca-jpxk6","openshift-multus/multus-additional-cni-plugins-5shsp","openshift-ovn-kubernetes/ovnkube-node-5w4ll"]
Jul 31 14:47:59.037335 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.037304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.038853 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.038824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.041316 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.041160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.042374 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042345    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.042491 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042434    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb6ml\""
Jul 31 14:47:59.042584 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042345    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.042643 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042577    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:47:59.042725 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042710    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:47:59.042766 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.042859 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.042843    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:47:59.043401 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043379    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.043546 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043526    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.043611 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043600    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:47:59.043673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043648    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k5ljw\""
Jul 31 14:47:59.043924 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043902    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.044000 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.043956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:47:59.046209 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.046187    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.046513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.046488    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.046691 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.046653    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-79fcl\""
Jul 31 14:47:59.046801 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.046725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.047351 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.047302    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:47:59.047642 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.047623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.047753 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.047705    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:47:59.049661 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.049638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:47:59.049768 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.049722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:47:59.050036 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.050011    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.050145 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.050043    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wmt65\""
Jul 31 14:47:59.052081 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.052026    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.052180 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.052105    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:47:59.052283 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.052258    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:47:59.052407 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.052353    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qfl5j\""
Jul 31 14:47:59.052771 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.052749    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:47:59.054073 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21ec7f6-eff2-4c88-8394-dca2df93ad25-host-slash\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.054179 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rp6\" (UniqueName: \"kubernetes.io/projected/a21ec7f6-eff2-4c88-8394-dca2df93ad25-kube-api-access-j2rp6\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.054179 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-system-cni-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054179 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-cnibin\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054191    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mxv\" (UniqueName: \"kubernetes.io/projected/50c437ff-669d-4c9f-aa01-039af0c80380-kube-api-access-p7mxv\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.054338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d99871e8-7e77-436c-8d5b-56dbcac433b4-hosts-file\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.054338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99871e8-7e77-436c-8d5b-56dbcac433b4-tmp-dir\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.054338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-os-release\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a21ec7f6-eff2-4c88-8394-dca2df93ad25-iptables-alerter-script\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrls\" (UniqueName: \"kubernetes.io/projected/56403f34-c078-4c81-805c-7882ec9e5284-kube-api-access-fzrls\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4w6k\" (UniqueName: \"kubernetes.io/projected/d99871e8-7e77-436c-8d5b-56dbcac433b4-kube-api-access-v4w6k\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-binary-copy\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c437ff-669d-4c9f-aa01-039af0c80380-host\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.054573 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.054570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50c437ff-669d-4c9f-aa01-039af0c80380-serviceca\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.055553 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.055355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.060075 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.059871    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:47:59.060075 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.060032    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xpgrz\""
Jul 31 14:47:59.063989 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.063923    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:47:59.066349 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.066205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.070469 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.070244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.071314 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.071291    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:47:59.072245 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.071509    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.072245 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.071823    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-88g7r\""
Jul 31 14:47:59.072245 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.072010    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.072804 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.072792    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.073579 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.073287    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nrwvf\""
Jul 31 14:47:59.073834 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.073755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.074968 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.074758    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.076380 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.076358    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:47:59.076933 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.076911    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:47:59.077278 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.077262    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:47:59.077546 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.077517    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:47:59.077773 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.077747    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7j449\""
Jul 31 14:47:59.078139 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.078118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:47:59.078463 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.078416    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:47:59.091119 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.091089    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ptjg"
Jul 31 14:47:59.101477 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.101056    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6ptjg"
Jul 31 14:47:59.143525 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.143486    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:47:59.155665 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-sys-fs\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.155866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-run\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.155866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.155866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-binary-copy\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.155866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-system-cni-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.155866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-cnibin\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scgb\" (UniqueName: \"kubernetes.io/projected/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-kube-api-access-7scgb\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-os-release\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.155985    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-kubelet\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-modprobe-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-registration-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.156101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-bin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-etc-kubernetes\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-systemd-units\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-config\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-kubelet-config\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-dbus\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmht\" (UniqueName: \"kubernetes.io/projected/380b53c0-18c5-44e1-b46d-3a0f59e5f975-kube-api-access-hwmht\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156353 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-host\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-bin\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-netd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156466    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a21ec7f6-eff2-4c88-8394-dca2df93ad25-iptables-alerter-script\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrls\" (UniqueName: \"kubernetes.io/projected/56403f34-c078-4c81-805c-7882ec9e5284-kube-api-access-fzrls\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-daemon-config\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8qp\" (UniqueName: \"kubernetes.io/projected/de489f2a-dbf3-4972-a8f1-84c98463eac5-kube-api-access-hd8qp\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50c437ff-669d-4c9f-aa01-039af0c80380-serviceca\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-binary-copy\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rp6\" (UniqueName: \"kubernetes.io/projected/a21ec7f6-eff2-4c88-8394-dca2df93ad25-kube-api-access-j2rp6\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d99871e8-7e77-436c-8d5b-56dbcac433b4-hosts-file\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.156650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-netns\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-conf-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-system-cni-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-sys\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156714    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-systemd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-cnibin\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-ovn\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.156789    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b77bb15f-e387-4e54-a709-c306d8e35922-konnectivity-ca\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.157321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-var-lib-kubelet\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-tuned\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a21ec7f6-eff2-4c88-8394-dca2df93ad25-iptables-alerter-script\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-kubelet\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-var-lib-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d99871e8-7e77-436c-8d5b-56dbcac433b4-hosts-file\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4w6k\" (UniqueName: \"kubernetes.io/projected/d99871e8-7e77-436c-8d5b-56dbcac433b4-kube-api-access-v4w6k\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157521    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-socket-dir-parent\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-multus-certs\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-conf\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-slash\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.157693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157661    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b77bb15f-e387-4e54-a709-c306d8e35922-agent-certs\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-kubernetes\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-env-overrides\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99871e8-7e77-436c-8d5b-56dbcac433b4-tmp-dir\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-systemd\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-lib-modules\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.157994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovn-node-metrics-cert\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158021    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvfg\" (UniqueName: \"kubernetes.io/projected/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kube-api-access-pxvfg\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysconfig\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158137 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xml\" (UniqueName: \"kubernetes.io/projected/01a2681f-b49a-48f7-9e97-4eefda47df6b-kube-api-access-49xml\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158216    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50c437ff-669d-4c9f-aa01-039af0c80380-serviceca\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-node-log\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-log-socket\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cni-binary-copy\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-hostroot\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c437ff-669d-4c9f-aa01-039af0c80380-host\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.158471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158444    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21ec7f6-eff2-4c88-8394-dca2df93ad25-host-slash\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-system-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-tmp\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mxv\" (UniqueName: \"kubernetes.io/projected/50c437ff-669d-4c9f-aa01-039af0c80380-kube-api-access-p7mxv\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158599    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.158788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-k8s-cni-cncf-io\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.159090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-netns\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.159090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50c437ff-669d-4c9f-aa01-039af0c80380-host\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.159090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.158952    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a21ec7f6-eff2-4c88-8394-dca2df93ad25-host-slash\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.159090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.159291 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99871e8-7e77-436c-8d5b-56dbcac433b4-tmp-dir\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.159291 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-script-lib\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.159477 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.159549 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56403f34-c078-4c81-805c-7882ec9e5284-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.159655 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-os-release\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.159746 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159691    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-socket-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.159804 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cnibin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.159804 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159794    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-multus\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.159906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-etc-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.159906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159857    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.159906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-device-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.160062 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.159957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56403f34-c078-4c81-805c-7882ec9e5284-os-release\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.172003 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.171962    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:47:59.177135 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.176595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mxv\" (UniqueName: \"kubernetes.io/projected/50c437ff-669d-4c9f-aa01-039af0c80380-kube-api-access-p7mxv\") pod \"node-ca-jpxk6\" (UID: \"50c437ff-669d-4c9f-aa01-039af0c80380\") " pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.177135 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.177030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrls\" (UniqueName: \"kubernetes.io/projected/56403f34-c078-4c81-805c-7882ec9e5284-kube-api-access-fzrls\") pod \"multus-additional-cni-plugins-5shsp\" (UID: \"56403f34-c078-4c81-805c-7882ec9e5284\") " pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.177135 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.177085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rp6\" (UniqueName: \"kubernetes.io/projected/a21ec7f6-eff2-4c88-8394-dca2df93ad25-kube-api-access-j2rp6\") pod \"iptables-alerter-5psbp\" (UID: \"a21ec7f6-eff2-4c88-8394-dca2df93ad25\") " pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.178036 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.177982    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4w6k\" (UniqueName: \"kubernetes.io/projected/d99871e8-7e77-436c-8d5b-56dbcac433b4-kube-api-access-v4w6k\") pod \"node-resolver-94z8k\" (UID: \"d99871e8-7e77-436c-8d5b-56dbcac433b4\") " pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.260337 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-conf-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260350    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-sys\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-conf-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-systemd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-sys\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-ovn\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260514    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-systemd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260540 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b77bb15f-e387-4e54-a709-c306d8e35922-konnectivity-ca\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260589    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-var-lib-kubelet\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260612    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-tuned\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-kubelet\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-var-lib-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-socket-dir-parent\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-multus-certs\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-conf\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-slash\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b77bb15f-e387-4e54-a709-c306d8e35922-agent-certs\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-kubernetes\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-env-overrides\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-systemd\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-lib-modules\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovn-node-metrics-cert\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.260941 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260916    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-multus-certs\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvfg\" (UniqueName: \"kubernetes.io/projected/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kube-api-access-pxvfg\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysconfig\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260992    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysconfig\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-conf\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-slash\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b77bb15f-e387-4e54-a709-c306d8e35922-konnectivity-ca\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261425    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-var-lib-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-ovn\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-kubelet\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.260687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-var-lib-kubelet\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261545    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-systemd\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261601    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-kubernetes\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xml\" (UniqueName: \"kubernetes.io/projected/01a2681f-b49a-48f7-9e97-4eefda47df6b-kube-api-access-49xml\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261841    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-node-log\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.261870 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-log-socket\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cni-binary-copy\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-hostroot\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261939    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-system-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-tmp\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.261992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-k8s-cni-cncf-io\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-netns\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-script-lib\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-socket-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cnibin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-env-overrides\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-multus\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-multus\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-etc-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.262715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-device-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-sys-fs\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-run\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262343    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262426    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scgb\" (UniqueName: \"kubernetes.io/projected/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-kube-api-access-7scgb\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-node-log\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-os-release\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262496    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-kubelet\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262518    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-os-release\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-modprobe-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-registration-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-bin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262615    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-etc-kubernetes\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262629    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-modprobe-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262641    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-systemd-units\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-run\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262805    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-systemd-units\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.263606 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-log-socket\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262842    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-sysctl-d\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-etc-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.262954    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-lib-modules\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-device-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-sys-fs\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.262989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.263131    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:47:59.763095093 +0000 UTC m=+3.097215010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-netns\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263214    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-hostroot\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-system-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263277    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cni-binary-copy\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-registration-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.263367    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263376    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-run-openvswitch\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.264689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-cni-bin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.263437    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:47:59.763418592 +0000 UTC m=+3.097538508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-k8s-cni-cncf-io\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-cnibin\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-var-lib-kubelet\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-config\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-socket-dir\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-kubelet-config\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263705    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-dbus\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmht\" (UniqueName: \"kubernetes.io/projected/380b53c0-18c5-44e1-b46d-3a0f59e5f975-kube-api-access-hwmht\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-host\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263801    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-script-lib\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263845    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-etc-kubernetes\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-bin\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-netd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-daemon-config\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.265458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.263979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8qp\" (UniqueName: \"kubernetes.io/projected/de489f2a-dbf3-4972-a8f1-84c98463eac5-kube-api-access-hd8qp\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-kubelet-config\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-netns\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-etc-tuned\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264140    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94357ff7-eec1-4ce0-b983-f574a7204766-dbus\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovnkube-config\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264171    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01a2681f-b49a-48f7-9e97-4eefda47df6b-host\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-host-run-netns\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-bin\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264277    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de489f2a-dbf3-4972-a8f1-84c98463eac5-host-cni-netd\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264352    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-cni-dir\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-socket-dir-parent\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/380b53c0-18c5-44e1-b46d-3a0f59e5f975-multus-daemon-config\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.266348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.264966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de489f2a-dbf3-4972-a8f1-84c98463eac5-ovn-node-metrics-cert\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.267105 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.266609    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a2681f-b49a-48f7-9e97-4eefda47df6b-tmp\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.270411 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.270378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b77bb15f-e387-4e54-a709-c306d8e35922-agent-certs\") pod \"konnectivity-agent-rgwjh\" (UID: \"b77bb15f-e387-4e54-a709-c306d8e35922\") " pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.274008 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.273971    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:47:59.274008 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.274003    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:47:59.274206 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.274019    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:47:59.274206 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.274109    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:47:59.774086426 +0000 UTC m=+3.108206347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:47:59.276724 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.276639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8qp\" (UniqueName: \"kubernetes.io/projected/de489f2a-dbf3-4972-a8f1-84c98463eac5-kube-api-access-hd8qp\") pod \"ovnkube-node-5w4ll\" (UID: \"de489f2a-dbf3-4972-a8f1-84c98463eac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.277512 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.277487    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xml\" (UniqueName: \"kubernetes.io/projected/01a2681f-b49a-48f7-9e97-4eefda47df6b-kube-api-access-49xml\") pod \"tuned-lmgcm\" (UID: \"01a2681f-b49a-48f7-9e97-4eefda47df6b\") " pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.278451 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.278430    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvfg\" (UniqueName: \"kubernetes.io/projected/ff536b6e-edc0-4771-8a4d-5068ecfaa2bd-kube-api-access-pxvfg\") pod \"aws-ebs-csi-driver-node-ctlck\" (UID: \"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.280041 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.280005    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scgb\" (UniqueName: \"kubernetes.io/projected/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-kube-api-access-7scgb\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.285768 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.285744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmht\" (UniqueName: \"kubernetes.io/projected/380b53c0-18c5-44e1-b46d-3a0f59e5f975-kube-api-access-hwmht\") pod \"multus-h72g8\" (UID: \"380b53c0-18c5-44e1-b46d-3a0f59e5f975\") " pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.357505 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.357465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5psbp"
Jul 31 14:47:59.369620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.369568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5shsp"
Jul 31 14:47:59.378743 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.378713    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94z8k"
Jul 31 14:47:59.387791 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.387763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jpxk6"
Jul 31 14:47:59.396854 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.396827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:47:59.407906 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.407876    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h72g8"
Jul 31 14:47:59.417015 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.416982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck"
Jul 31 14:47:59.425025 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.424979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lmgcm"
Jul 31 14:47:59.433423 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.433393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:47:59.726366 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.726271    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:59.767740 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.767696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:47:59.767936 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.767777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:47:59.767936 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.767837    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:47:59.767936 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.767913    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.767895799 +0000 UTC m=+4.102015700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:47:59.767936 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.767931    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:47:59.768082 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.767988    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.767969151 +0000 UTC m=+4.102089068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:47:59.869269 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:47:59.869223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:47:59.869444 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.869413    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:47:59.869497 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.869446    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:47:59.869497 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.869461    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:47:59.869586 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:47:59.869546    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.869522674 +0000 UTC m=+4.203642601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.103942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:00.103838    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:42:59 +0000 UTC" deadline="2028-01-11 15:08:42.00310292 +0000 UTC"
Jul 31 14:48:00.103942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:00.103880    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12696h20m41.899228172s"
Jul 31 14:48:00.235086 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:00.235046    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99871e8_7e77_436c_8d5b_56dbcac433b4.slice/crio-a019bf2a4c8bfc2bb5d5b1ae94de2471243a1462c7bf09b0905786f1610e112c WatchSource:0}: Error finding container a019bf2a4c8bfc2bb5d5b1ae94de2471243a1462c7bf09b0905786f1610e112c: Status 404 returned error can't find the container with id a019bf2a4c8bfc2bb5d5b1ae94de2471243a1462c7bf09b0905786f1610e112c
Jul 31 14:48:00.246138 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:00.246108    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde489f2a_dbf3_4972_a8f1_84c98463eac5.slice/crio-583cfe644cf23488ec63d557bbcd5a19b63eb1d3e2fd5d4059a7e843767a2613 WatchSource:0}: Error finding container 583cfe644cf23488ec63d557bbcd5a19b63eb1d3e2fd5d4059a7e843767a2613: Status 404 returned error can't find the container with id 583cfe644cf23488ec63d557bbcd5a19b63eb1d3e2fd5d4059a7e843767a2613
Jul 31 14:48:00.251690 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:00.251640    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77bb15f_e387_4e54_a709_c306d8e35922.slice/crio-3cbd5d21df653c151451025f2b91c465af251f8d1bfeedbe0df7bbdd501112d2 WatchSource:0}: Error finding container 3cbd5d21df653c151451025f2b91c465af251f8d1bfeedbe0df7bbdd501112d2: Status 404 returned error can't find the container with id 3cbd5d21df653c151451025f2b91c465af251f8d1bfeedbe0df7bbdd501112d2
Jul 31 14:48:00.254961 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:00.254937    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21ec7f6_eff2_4c88_8394_dca2df93ad25.slice/crio-3101954a2dd8a6841cee75b2666920c64ebe4f38705c33463ab6dc48497cf6d4 WatchSource:0}: Error finding container 3101954a2dd8a6841cee75b2666920c64ebe4f38705c33463ab6dc48497cf6d4: Status 404 returned error can't find the container with id 3101954a2dd8a6841cee75b2666920c64ebe4f38705c33463ab6dc48497cf6d4
Jul 31 14:48:00.258588 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:00.258559    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c437ff_669d_4c9f_aa01_039af0c80380.slice/crio-c3946024243e43418459e38035fd4cca3054fa8bffe6dca213c11dd6a053a846 WatchSource:0}: Error finding container c3946024243e43418459e38035fd4cca3054fa8bffe6dca213c11dd6a053a846: Status 404 returned error can't find the container with id c3946024243e43418459e38035fd4cca3054fa8bffe6dca213c11dd6a053a846
Jul 31 14:48:00.776733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:00.776600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:00.776733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:00.776692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:00.776986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.776805    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.776986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.776873    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:00.776986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.776895    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:02.776872903 +0000 UTC m=+6.110992820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.776986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.776939    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:48:02.776919751 +0000 UTC m=+6.111039655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:00.878181 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:00.878139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:00.878385 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.878323    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:00.878385 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.878346    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:00.878385 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.878363    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.878546 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:00.878432    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:48:02.878409811 +0000 UTC m=+6.212529720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:01.104916 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.104787    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:42:59 +0000 UTC" deadline="2028-03-13 09:18:57.971527071 +0000 UTC"
Jul 31 14:48:01.104916 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.104857    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14178h30m56.866674997s"
Jul 31 14:48:01.200038 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.200004    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:01.200214 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:01.200152    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:01.200612 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.200589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:01.200741 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:01.200712    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:01.200850 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.200794    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:01.200968 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:01.200901    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:01.214788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.213887    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" event={"ID":"caa64f70662f34d6016d67055768843e","Type":"ContainerStarted","Data":"681e5a8d788f34c0d4fef0ab250c16cd7f617a0014dbada1e791e985a951caa7"}
Jul 31 14:48:01.217935 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.217899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmgcm" event={"ID":"01a2681f-b49a-48f7-9e97-4eefda47df6b","Type":"ContainerStarted","Data":"31140f7fe8cff6db8f8864eb4966a596f52c495cdb39da3fce8c2ec5fb3d1112"}
Jul 31 14:48:01.223292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.223258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerStarted","Data":"ac826130bdfae368a43ffbaa13ebe1b78d36dfb40a02e4c73fac4b1e728fa70d"}
Jul 31 14:48:01.233722 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.233638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h72g8" event={"ID":"380b53c0-18c5-44e1-b46d-3a0f59e5f975","Type":"ContainerStarted","Data":"988e945e54e773adad42ecda090e10713fd1e43e39b8058c4b9890bfd6edfeb7"}
Jul 31 14:48:01.239398 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.239354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck" event={"ID":"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd","Type":"ContainerStarted","Data":"75a97d47b70afe49be224647674cf88196a1492ce39c66b6845489fa715d9cf8"}
Jul 31 14:48:01.246345 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.246303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpxk6" event={"ID":"50c437ff-669d-4c9f-aa01-039af0c80380","Type":"ContainerStarted","Data":"c3946024243e43418459e38035fd4cca3054fa8bffe6dca213c11dd6a053a846"}
Jul 31 14:48:01.253227 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.253184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94z8k" event={"ID":"d99871e8-7e77-436c-8d5b-56dbcac433b4","Type":"ContainerStarted","Data":"a019bf2a4c8bfc2bb5d5b1ae94de2471243a1462c7bf09b0905786f1610e112c"}
Jul 31 14:48:01.261343 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.261301    2575 generic.go:356] "Generic (PLEG): container finished" podID="0396405a5f8273b532964273d7d51214" containerID="1d4941793b48884b068b696d3fac7d0aefdf391a3fde9375640ea06af82754bb" exitCode=0
Jul 31 14:48:01.261538 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.261397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"0396405a5f8273b532964273d7d51214","Type":"ContainerDied","Data":"1d4941793b48884b068b696d3fac7d0aefdf391a3fde9375640ea06af82754bb"}
Jul 31 14:48:01.278760 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.278718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5psbp" event={"ID":"a21ec7f6-eff2-4c88-8394-dca2df93ad25","Type":"ContainerStarted","Data":"3101954a2dd8a6841cee75b2666920c64ebe4f38705c33463ab6dc48497cf6d4"}
Jul 31 14:48:01.283220 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.282923    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" podStartSLOduration=3.28289888 podStartE2EDuration="3.28289888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:01.236518225 +0000 UTC m=+4.570638152" watchObservedRunningTime="2026-07-31 14:48:01.28289888 +0000 UTC m=+4.617018806"
Jul 31 14:48:01.284655 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.284562    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rgwjh" event={"ID":"b77bb15f-e387-4e54-a709-c306d8e35922","Type":"ContainerStarted","Data":"3cbd5d21df653c151451025f2b91c465af251f8d1bfeedbe0df7bbdd501112d2"}
Jul 31 14:48:01.287527 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:01.287493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"583cfe644cf23488ec63d557bbcd5a19b63eb1d3e2fd5d4059a7e843767a2613"}
Jul 31 14:48:02.330124 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:02.330085    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"0396405a5f8273b532964273d7d51214","Type":"ContainerStarted","Data":"7f61c961a5598c0fcf91f5065fc3a5dfd3d1393587e080472367ce6aaf7f1439"}
Jul 31 14:48:02.354183 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:02.353457    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" podStartSLOduration=4.353432021 podStartE2EDuration="4.353432021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:02.352493497 +0000 UTC m=+5.686613425" watchObservedRunningTime="2026-07-31 14:48:02.353432021 +0000 UTC m=+5.687551949"
Jul 31 14:48:02.797116 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:02.797012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:02.797116 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:02.797078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:02.797380 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.797225    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:02.797380 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.797360    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:48:06.797333442 +0000 UTC m=+10.131453364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:02.797926 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.797903    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:02.798030 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.797967    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:06.797949601 +0000 UTC m=+10.132069506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:02.898161 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:02.898116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:02.898390 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.898319    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:02.898390 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.898345    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:02.898390 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.898358    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:02.898578 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:02.898427    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:48:06.898405208 +0000 UTC m=+10.232525111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:03.198770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:03.198648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:03.198955 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:03.198822    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:03.199274 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:03.199251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:03.199428 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:03.199368    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:03.199503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:03.199438    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:03.199559 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:03.199518    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:05.197862 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:05.197823    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:05.198522 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:05.197992    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:05.198522 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:05.198401    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:05.198522 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:05.198499    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:05.198742 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:05.198557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:05.198742 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:05.198633    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:06.833249 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:06.833140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:06.833249 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:06.833251    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:06.833931 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.833327    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:06.833931 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.833368    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:06.833931 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.833414    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.833389418 +0000 UTC m=+18.167509540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:06.833931 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.833439    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.833426804 +0000 UTC m=+18.167546709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:06.934467 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:06.934424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:06.934664 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.934584    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:06.934664 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.934622    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:06.934664 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.934637    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:06.934783 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:06.934687    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:48:14.93467229 +0000 UTC m=+18.268792190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:07.197678 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:07.197578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:07.197833 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:07.197575    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:07.197833 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:07.197721    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:07.197833 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:07.197809    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:07.197833 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:07.197581    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:07.198026 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:07.197912    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:09.195947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:09.196080    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:09.196201    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:09.196304    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:09.196344    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:09.196717 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:09.196396    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:11.196969 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:11.196926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:11.197435 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:11.196926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:11.197435 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:11.197062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:11.197435 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:11.196935    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:11.197435 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:11.197190    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:11.197435 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:11.197238    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:13.196265 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:13.196228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:13.196720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:13.196274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:13.196720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:13.196228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:13.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:13.196369    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:13.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:13.196466    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:13.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:13.196565    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:14.899923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:14.900038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:14.900119    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:14.900117    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:14.900172    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:30.900158092 +0000 UTC m=+34.234277992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:14.900523 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:14.900187    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:48:30.900179987 +0000 UTC m=+34.234299886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:15.001370 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:15.001332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:15.001548 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.001496    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:15.001548 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.001519    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:15.001548 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.001529    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:15.001738 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.001593    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:48:31.001571747 +0000 UTC m=+34.335691671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:15.197059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:15.196965    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:15.197059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:15.196996    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:15.197059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:15.196986    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:15.197332 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.197119    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:15.197332 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.197251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:15.197440 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:15.197403    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:17.197842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:17.197974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:17.197995    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:17.197976    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:17.198090    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:17.198311 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:17.198172    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:19.196221 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:19.196185    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:19.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:19.196302    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:19.196720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:19.196203    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:19.196720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:19.196326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:19.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:19.196438    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:19.196720 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:19.196494    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:20.369149 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.368697    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lmgcm" event={"ID":"01a2681f-b49a-48f7-9e97-4eefda47df6b","Type":"ContainerStarted","Data":"ef253eb4b9f2cd68bf73fbdd4de0ec43a7cbd8d218d6435c101270570497878c"}
Jul 31 14:48:20.370325 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.370297    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="0edb246e307609d52b3256b1cccc90365c6ead9ba8cd5fe1c0bffc04454fe1db" exitCode=0
Jul 31 14:48:20.370439 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.370358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"0edb246e307609d52b3256b1cccc90365c6ead9ba8cd5fe1c0bffc04454fe1db"}
Jul 31 14:48:20.371936 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.371814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h72g8" event={"ID":"380b53c0-18c5-44e1-b46d-3a0f59e5f975","Type":"ContainerStarted","Data":"43b1fead16f5632e32afe8aef8d17ac83e50f62358e450d00bac412c1ac4ddb2"}
Jul 31 14:48:20.375619 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.375578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck" event={"ID":"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd","Type":"ContainerStarted","Data":"77132d47b71792fdd0f36505929770ea0f7dfb72dec16c7af3b2be627af24c83"}
Jul 31 14:48:20.377210 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.377183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jpxk6" event={"ID":"50c437ff-669d-4c9f-aa01-039af0c80380","Type":"ContainerStarted","Data":"365d65ae9e60e4a00a6d47b59475609c16e9ae567d112862146e24bd343b2249"}
Jul 31 14:48:20.378484 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.378464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94z8k" event={"ID":"d99871e8-7e77-436c-8d5b-56dbcac433b4","Type":"ContainerStarted","Data":"ce5c8013eadcc15c7225050f03b9a17d55b405eb4ce982eec06af6db00123039"}
Jul 31 14:48:20.379967 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.379942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rgwjh" event={"ID":"b77bb15f-e387-4e54-a709-c306d8e35922","Type":"ContainerStarted","Data":"de8474ec1ef5f2db22ea594f4c47bbf5c42f9fae1ac632ac6772c4d141b36b44"}
Jul 31 14:48:20.383217 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"6ec4aba8dc1870748f4e82fd629e77c65e945aed4916c421c2db0d7b10478a29"}
Jul 31 14:48:20.383329 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"1bdddcc8353eeb82f3466854ab971be58f1f52ef11cad4e8dfe8bc56ad921608"}
Jul 31 14:48:20.383329 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383326    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"69acbd1a02faa6d2d73383094482026e42f2cb4dd9af7940a0f700518ac6127c"}
Jul 31 14:48:20.383428 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"cfb3e6a4a0a8627dd4c01be29dad6b57de41345886228aeec6de67ff0ba21682"}
Jul 31 14:48:20.383428 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383402    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"d42ae584943b114e0e14d17d9747b54b803d5cfef8f041985e3cd5560c4f3fdf"}
Jul 31 14:48:20.383503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.383437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"65c0a8b21071e2797d241740edd6bbeacea442b5a7bc52fe529039c3cc0e7212"}
Jul 31 14:48:20.387615 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.387551    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lmgcm" podStartSLOduration=3.119691602 podStartE2EDuration="22.38753879s" totalImagesPullingTime="19.267847188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:20.387266131 +0000 UTC m=+23.721386052" watchObservedRunningTime="2026-07-31 14:48:20.38753879 +0000 UTC m=+23.721658780"
Jul 31 14:48:20.403023 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.402971    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-94z8k" podStartSLOduration=4.129651943 podStartE2EDuration="23.402954859s" totalImagesPullingTime="19.273302916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:20.402307567 +0000 UTC m=+23.736427488" watchObservedRunningTime="2026-07-31 14:48:20.402954859 +0000 UTC m=+23.737074781"
Jul 31 14:48:20.418450 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.418395    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jpxk6" podStartSLOduration=4.162379974 podStartE2EDuration="23.418376098s" totalImagesPullingTime="19.255996124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:20.41798232 +0000 UTC m=+23.752102241" watchObservedRunningTime="2026-07-31 14:48:20.418376098 +0000 UTC m=+23.752496020"
Jul 31 14:48:20.437773 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.437708    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-h72g8" podStartSLOduration=3.129368144 podStartE2EDuration="22.437688581s" totalImagesPullingTime="19.308320437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:20.437173408 +0000 UTC m=+23.771293332" watchObservedRunningTime="2026-07-31 14:48:20.437688581 +0000 UTC m=+23.771808505"
Jul 31 14:48:20.454356 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.454305    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rgwjh" podStartSLOduration=7.907453084 podStartE2EDuration="23.454287124s" totalImagesPullingTime="15.54683404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:20.45358115 +0000 UTC m=+23.787701071" watchObservedRunningTime="2026-07-31 14:48:20.454287124 +0000 UTC m=+23.788407046"
Jul 31 14:48:20.559567 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.559529    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:20.623280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.623200    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:20.831707 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.831666    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:20.908894 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:20.908805    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:21.198502 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.198462    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:21.198751 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:21.198627    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:21.199060 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.199040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:21.199172 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:21.199150    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:21.199239 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.199223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:21.199312 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:21.199296    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:21.302799 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.302766    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:21.355810 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.355775    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:48:21.356548 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.356515    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:48:21.387286 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.387245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5psbp" event={"ID":"a21ec7f6-eff2-4c88-8394-dca2df93ad25","Type":"ContainerStarted","Data":"83fbbac35df2f240dbfa5a1410e81ad842c504c1e199c327a875c14b006c3ea6"}
Jul 31 14:48:21.389544 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.389514    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck" event={"ID":"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd","Type":"ContainerStarted","Data":"a27c8fe83abeedc4066159c17e33ee1c049c813e04f9c9bb8a8ffb1e3290364e"}
Jul 31 14:48:21.389922 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.389902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:48:21.390236 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.390218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rgwjh"
Jul 31 14:48:21.407648 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.407562    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5psbp" podStartSLOduration=8.868123779 podStartE2EDuration="24.407543119s" totalImagesPullingTime="15.53941934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.407194141 +0000 UTC m=+24.741314065" watchObservedRunningTime="2026-07-31 14:48:21.407543119 +0000 UTC m=+24.741663041"
Jul 31 14:48:21.545508 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.545361    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:21.648378 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.648342    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:21.717422 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:21.717383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:22.122354 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:22.122215    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:21.302791717Z","UUID":"9f135492-348b-4566-a794-c1c3aee69579","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:22.125001 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:22.124977    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:22.125145 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:22.125024    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:23.115912 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.115804    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:23.196813 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.196780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:23.196996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.196780    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:23.196996 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:23.196936    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:23.196996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.196784    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:23.196996 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:23.196986    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:23.197155 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:23.197106    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:23.218922 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.218892    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:23.403033 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.402940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"f007cd6c2f66e70eda634ff1ce8462ec74270de87aab9521787ddd5d69a1ea0c"}
Jul 31 14:48:23.405462 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.405433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck" event={"ID":"ff536b6e-edc0-4771-8a4d-5068ecfaa2bd","Type":"ContainerStarted","Data":"1e1d0cc9ace8a51e656c83cf2acd552e3453e61639cbf6d8b4fc3a497da9f2bd"}
Jul 31 14:48:23.432060 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:23.431998    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctlck" podStartSLOduration=3.478526776 podStartE2EDuration="25.431976393s" totalImagesPullingTime="21.953449617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:23.431289129 +0000 UTC m=+26.765409048" watchObservedRunningTime="2026-07-31 14:48:23.431976393 +0000 UTC m=+26.766096319"
Jul 31 14:48:24.090458 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:24.090418    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:24.188902 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:24.188828    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:24.261971 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:24.261937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.196814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.196819    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:25.197186    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:25.197250    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.196874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:25.197991 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:25.197354    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:25.412541 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.412505    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="b1262c407c5d8b9cb0007eed9c961f58c53b13375f5ef0e8fb2e32495cb6ddee" exitCode=0
Jul 31 14:48:25.412748 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.412554    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"b1262c407c5d8b9cb0007eed9c961f58c53b13375f5ef0e8fb2e32495cb6ddee"}
Jul 31 14:48:25.416154 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.416133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" event={"ID":"de489f2a-dbf3-4972-a8f1-84c98463eac5","Type":"ContainerStarted","Data":"49bb4f3a64afafe4fc2730442c3c958641dce63bc8b8aa55d775660f1a4bc7db"}
Jul 31 14:48:25.416437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.416420    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:25.416510 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.416491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:25.432802 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.432777    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:25.475429 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:25.475377    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll" podStartSLOduration=8.135226168 podStartE2EDuration="27.475361708s" totalImagesPullingTime="19.34013554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:25.473824675 +0000 UTC m=+28.807944594" watchObservedRunningTime="2026-07-31 14:48:25.475361708 +0000 UTC m=+28.809481653"
Jul 31 14:48:26.012096 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:26.012025    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:26.419289 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:26.419207    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:26.437090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:26.436914    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:27.197787 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.197752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:27.198000 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.197850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:27.198000 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.197896    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:27.198000 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.197922    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:27.198000 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.197974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:27.198208 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.198062    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:27.328846 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.328817    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xcfwb"]
Jul 31 14:48:27.331947 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.331926    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nt9j7"]
Jul 31 14:48:27.335218 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.335143    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dt4ln"]
Jul 31 14:48:27.422820 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.422789    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="da88d5fb849acef93bd2db6275f7c3850212fab8802d341c1898d25758f07375" exitCode=0
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.422913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"da88d5fb849acef93bd2db6275f7c3850212fab8802d341c1898d25758f07375"}
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.422998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.422999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.423107    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:27.423118    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.423182    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:27.423292 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:27.423285    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:28.261371 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:28.261339    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:28.359206 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:28.359171    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:29.197082 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:29.196899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:29.197518 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:29.196899    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:29.197518 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:29.197212    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:29.197518 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:29.196985    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:29.197518 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:29.197253    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:29.197518 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:29.197387    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:29.429576 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:29.429535    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="5f6694b528f2de389900aae0530b3be7b32a1d28b910e35e0d3cbf483fc3fe35" exitCode=0
Jul 31 14:48:29.429767 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:29.429628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"5f6694b528f2de389900aae0530b3be7b32a1d28b910e35e0d3cbf483fc3fe35"}
Jul 31 14:48:30.914544 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:30.914507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:30.914998 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:30.914590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:30.914998 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:30.914698    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:30.914998 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:30.914738    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:30.914998 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:30.914762    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs podName:61802f40-474e-4d7d-94ef-a3bfc53e6a6f nodeName:}" failed. No retries permitted until 2026-07-31 14:49:02.91474429 +0000 UTC m=+66.248864209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs") pod "network-metrics-daemon-xcfwb" (UID: "61802f40-474e-4d7d-94ef-a3bfc53e6a6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:30.914998 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:30.914792    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret podName:94357ff7-eec1-4ce0-b983-f574a7204766 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:02.914777023 +0000 UTC m=+66.248896923 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret") pod "global-pull-secret-syncer-nt9j7" (UID: "94357ff7-eec1-4ce0-b983-f574a7204766") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:31.015997 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:31.015956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:31.016244 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.016103    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:31.016244 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.016123    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:31.016244 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.016135    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4zgcc for pod openshift-network-diagnostics/network-check-target-dt4ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:31.016244 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.016194    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc podName:0465a7ba-0b2d-4366-bbdf-27bacfa77a2c nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.016179572 +0000 UTC m=+66.350299472 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4zgcc" (UniqueName: "kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc") pod "network-check-target-dt4ln" (UID: "0465a7ba-0b2d-4366-bbdf-27bacfa77a2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:31.196454 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:31.196361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:31.196737 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:31.196361    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:31.196737 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.196498    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xcfwb" podUID="61802f40-474e-4d7d-94ef-a3bfc53e6a6f"
Jul 31 14:48:31.196737 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:31.196367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:31.196737 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.196557    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt4ln" podUID="0465a7ba-0b2d-4366-bbdf-27bacfa77a2c"
Jul 31 14:48:31.196737 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:31.196638    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nt9j7" podUID="94357ff7-eec1-4ce0-b983-f574a7204766"
Jul 31 14:48:32.489552 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.489468    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeReady"
Jul 31 14:48:32.490019 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.489612    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:32.532436 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.532398    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:48:32.534840 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.534815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.537481 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.537450    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4fm2\""
Jul 31 14:48:32.537716 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.537698    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 14:48:32.537829 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.537797    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 14:48:32.537893 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.537822    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 14:48:32.543662 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.543640    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 14:48:32.544939 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.544917    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zx94w"]
Jul 31 14:48:32.546753 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.546734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.550506 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.550479    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-czhw9"]
Jul 31 14:48:32.552646 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.552578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.553529 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.553510    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:48:32.553811 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.553785    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:48:32.553922 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.553904    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:48:32.553994 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.553979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xps48\""
Jul 31 14:48:32.558418 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.558397    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:48:32.558614 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.558578    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:48:32.559781 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.559422    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-46zx7\""
Jul 31 14:48:32.559781 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.559687    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:48:32.566334 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.566311    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zx94w"]
Jul 31 14:48:32.573517 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.573462    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czhw9"]
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-cert\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmmz\" (UniqueName: \"kubernetes.io/projected/558d0e42-9039-431d-b66d-51770b2ef959-kube-api-access-msmmz\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cch6f\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.627997    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.628089 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.628051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.628733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.628109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558d0e42-9039-431d-b66d-51770b2ef959-tmp-dir\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.628733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.628150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2dx\" (UniqueName: \"kubernetes.io/projected/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-kube-api-access-6t2dx\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.628733 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.628187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/558d0e42-9039-431d-b66d-51770b2ef959-config-volume\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.728813 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.728813 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-cert\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmmz\" (UniqueName: \"kubernetes.io/projected/558d0e42-9039-431d-b66d-51770b2ef959-kube-api-access-msmmz\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:32.728949    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:32.728967    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdf7c5b-2wgc5: secret "image-registry-tls" not found
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.728983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cch6f\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729059 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:32.729033    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls podName:87744021-57ed-452d-8187-173d2f849f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.22901039 +0000 UTC m=+36.563130312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls") pod "image-registry-55fdf7c5b-2wgc5" (UID: "87744021-57ed-452d-8187-173d2f849f42") : secret "image-registry-tls" not found
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558d0e42-9039-431d-b66d-51770b2ef959-tmp-dir\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729158    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2dx\" (UniqueName: \"kubernetes.io/projected/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-kube-api-access-6t2dx\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/558d0e42-9039-431d-b66d-51770b2ef959-config-volume\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.729405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.730205 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558d0e42-9039-431d-b66d-51770b2ef959-tmp-dir\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.730205 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.729850    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.730205 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:32.729881    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:32.730205 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:32.729929    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls podName:558d0e42-9039-431d-b66d-51770b2ef959 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.229912138 +0000 UTC m=+36.564032045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls") pod "dns-default-czhw9" (UID: "558d0e42-9039-431d-b66d-51770b2ef959") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:32.730427 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.730373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/558d0e42-9039-431d-b66d-51770b2ef959-config-volume\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.731330 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.730961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.731330 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.731211    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.734420 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.734391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.734420 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.734408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-cert\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.735082 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.735051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.744012 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.743936    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:32.744855 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.744807    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cch6f\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.746957 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.746922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:32.747085 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.747009    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2dx\" (UniqueName: \"kubernetes.io/projected/22d806cf-8c2e-4a36-a22f-f7e32bb1f619-kube-api-access-6t2dx\") pod \"ingress-canary-zx94w\" (UID: \"22d806cf-8c2e-4a36-a22f-f7e32bb1f619\") " pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:32.747532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.747512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmmz\" (UniqueName: \"kubernetes.io/projected/558d0e42-9039-431d-b66d-51770b2ef959-kube-api-access-msmmz\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:32.847446 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.845498    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:32.859194 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:32.859158    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zx94w"
Jul 31 14:48:33.151067 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.151022    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"]
Jul 31 14:48:33.178863 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.178823    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"]
Jul 31 14:48:33.179030 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.178971    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"
Jul 31 14:48:33.182936 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.182899    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:48:33.183097 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.182950    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ng86f\""
Jul 31 14:48:33.183097 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.182951    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:48:33.196495 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.196010    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:48:33.196495 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.196050    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:48:33.196495 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.196052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:48:33.199577 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.199544    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2kc8\""
Jul 31 14:48:33.199756 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.199662    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:48:33.199841 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.199819    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:48:33.200061 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.199897    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4v8n\""
Jul 31 14:48:33.233404 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.233373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:33.233575 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.233445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:33.233575 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.233537    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:33.233575 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.233538    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:33.233575 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.233558    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdf7c5b-2wgc5: secret "image-registry-tls" not found
Jul 31 14:48:33.233731 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.233617    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls podName:558d0e42-9039-431d-b66d-51770b2ef959 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.233579881 +0000 UTC m=+37.567699781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls") pod "dns-default-czhw9" (UID: "558d0e42-9039-431d-b66d-51770b2ef959") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:33.233731 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.233636    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls podName:87744021-57ed-452d-8187-173d2f849f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.233627178 +0000 UTC m=+37.567747078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls") pod "image-registry-55fdf7c5b-2wgc5" (UID: "87744021-57ed-452d-8187-173d2f849f42") : secret "image-registry-tls" not found
Jul 31 14:48:33.258262 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.258222    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:33.333984 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.333935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj9tb\" (UniqueName: \"kubernetes.io/projected/a97a064f-de87-4f33-9387-5384c5967a2e-kube-api-access-mj9tb\") pod \"network-check-source-65bdbbff45-d724s\" (UID: \"a97a064f-de87-4f33-9387-5384c5967a2e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"
Jul 31 14:48:33.396276 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.396238    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:33.435335 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.435243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj9tb\" (UniqueName: \"kubernetes.io/projected/a97a064f-de87-4f33-9387-5384c5967a2e-kube-api-access-mj9tb\") pod \"network-check-source-65bdbbff45-d724s\" (UID: \"a97a064f-de87-4f33-9387-5384c5967a2e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"
Jul 31 14:48:33.513764 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.513732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj9tb\" (UniqueName: \"kubernetes.io/projected/a97a064f-de87-4f33-9387-5384c5967a2e-kube-api-access-mj9tb\") pod \"network-check-source-65bdbbff45-d724s\" (UID: \"a97a064f-de87-4f33-9387-5384c5967a2e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"
Jul 31 14:48:33.609409 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.609367    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:33.665188 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.665152    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2w2zm"]
Jul 31 14:48:33.684655 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.684619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.687795 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.687708    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:48:33.687795 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.687715    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:48:33.688107 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.688085    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:48:33.688200 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.688118    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:48:33.688520 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.688496    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfnvf\""
Jul 31 14:48:33.688765 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.688743    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2w2zm"]
Jul 31 14:48:33.790519 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.790479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"
Jul 31 14:48:33.838492 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.838453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebf922ca-e856-43c9-8e88-899512ec8fbe-crio-socket\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.838727 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.838502    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.838727 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.838531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bb9c\" (UniqueName: \"kubernetes.io/projected/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-api-access-4bb9c\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.838727 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.838585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf922ca-e856-43c9-8e88-899512ec8fbe-data-volume\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.838727 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.838651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939469 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf922ca-e856-43c9-8e88-899512ec8fbe-data-volume\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939469 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939718 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939525    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebf922ca-e856-43c9-8e88-899512ec8fbe-crio-socket\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939718 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939718 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bb9c\" (UniqueName: \"kubernetes.io/projected/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-api-access-4bb9c\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939718 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.939584    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:33.939718 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.939682    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:33.939986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.939754    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.439732259 +0000 UTC m=+37.773852183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:33.939986 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:33.939774    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.439764363 +0000 UTC m=+37.773884266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:33.939986 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebf922ca-e856-43c9-8e88-899512ec8fbe-data-volume\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.939986 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.939826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ebf922ca-e856-43c9-8e88-899512ec8fbe-crio-socket\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:33.950185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:33.950151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bb9c\" (UniqueName: \"kubernetes.io/projected/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-api-access-4bb9c\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:34.242982 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.242885    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:34.243155 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.243004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:34.243155 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.243052    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:34.243155 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.243074    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdf7c5b-2wgc5: secret "image-registry-tls" not found
Jul 31 14:48:34.243155 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.243117    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:34.243155 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.243140    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls podName:87744021-57ed-452d-8187-173d2f849f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:36.243120091 +0000 UTC m=+39.577240014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls") pod "image-registry-55fdf7c5b-2wgc5" (UID: "87744021-57ed-452d-8187-173d2f849f42") : secret "image-registry-tls" not found
Jul 31 14:48:34.243345 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.243164    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls podName:558d0e42-9039-431d-b66d-51770b2ef959 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:36.243148869 +0000 UTC m=+39.577268787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls") pod "dns-default-czhw9" (UID: "558d0e42-9039-431d-b66d-51770b2ef959") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:34.415669 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.415627    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:34.444334 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.444304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:34.444482 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.444384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:34.444482 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.444464    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:34.444563 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.444504    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:34.444563 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.444529    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:35.444514569 +0000 UTC m=+38.778634469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:34.444678 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:34.444566    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:35.444552973 +0000 UTC m=+38.778672874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:34.502985 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.502897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:34.639544 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.639364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-d724s"]
Jul 31 14:48:34.643493 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.643466    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zx94w"]
Jul 31 14:48:34.708280 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:34.708193    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97a064f_de87_4f33_9387_5384c5967a2e.slice/crio-5da2f81f1f3e3bd130d4b3b66dc4b3c9b8d4163e36e1987f852ab10275111921 WatchSource:0}: Error finding container 5da2f81f1f3e3bd130d4b3b66dc4b3c9b8d4163e36e1987f852ab10275111921: Status 404 returned error can't find the container with id 5da2f81f1f3e3bd130d4b3b66dc4b3c9b8d4163e36e1987f852ab10275111921
Jul 31 14:48:34.756391 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.756285    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94z8k_d99871e8-7e77-436c-8d5b-56dbcac433b4/dns-node-resolver/0.log"
Jul 31 14:48:34.835899 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.835871    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:34.932418 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:34.932385    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.104806 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.104780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.204736 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.204700    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.330634 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.330173    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"]
Jul 31 14:48:35.342265 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.342231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"
Jul 31 14:48:35.342452 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.342409    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"]
Jul 31 14:48:35.345575 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.345548    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:35.346348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.346323    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 14:48:35.346464 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.346402    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zdvzv\""
Jul 31 14:48:35.367647 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.367532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.448921 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.448882    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="30df30be3bde3bcbe169fc7c0c517befe265995c286b5a069d10b6e3af3e54cd" exitCode=0
Jul 31 14:48:35.449105 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.448989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"30df30be3bde3bcbe169fc7c0c517befe265995c286b5a069d10b6e3af3e54cd"}
Jul 31 14:48:35.451822 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.451797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:35.451945 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.451870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:35.451945 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.451923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446tc\" (UniqueName: \"kubernetes.io/projected/3b076be2-0dd9-4f62-bef3-e00ecb467b25-kube-api-access-446tc\") pod \"migrator-68bc745d87-d4jdn\" (UID: \"3b076be2-0dd9-4f62-bef3-e00ecb467b25\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"
Jul 31 14:48:35.452064 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:35.452041    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:35.452118 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:35.452085    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:37.452071065 +0000 UTC m=+40.786190964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:35.452416 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:35.452400    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:35.452474 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:35.452450    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:37.452437929 +0000 UTC m=+40.786557829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:35.452690 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.452660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s" event={"ID":"a97a064f-de87-4f33-9387-5384c5967a2e","Type":"ContainerStarted","Data":"5da2f81f1f3e3bd130d4b3b66dc4b3c9b8d4163e36e1987f852ab10275111921"}
Jul 31 14:48:35.454896 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.454826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zx94w" event={"ID":"22d806cf-8c2e-4a36-a22f-f7e32bb1f619","Type":"ContainerStarted","Data":"cf22ffdb41f9e7ad37cd55ad3dc218aaae7dbf786d79dcb8828e23d5e6929293"}
Jul 31 14:48:35.468440 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.468380    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.553221 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.553181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446tc\" (UniqueName: \"kubernetes.io/projected/3b076be2-0dd9-4f62-bef3-e00ecb467b25-kube-api-access-446tc\") pod \"migrator-68bc745d87-d4jdn\" (UID: \"3b076be2-0dd9-4f62-bef3-e00ecb467b25\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"
Jul 31 14:48:35.594996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.594920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-446tc\" (UniqueName: \"kubernetes.io/projected/3b076be2-0dd9-4f62-bef3-e00ecb467b25-kube-api-access-446tc\") pod \"migrator-68bc745d87-d4jdn\" (UID: \"3b076be2-0dd9-4f62-bef3-e00ecb467b25\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"
Jul 31 14:48:35.654969 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.654932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"
Jul 31 14:48:35.765954 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.765821    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.835081 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.835045    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn"]
Jul 31 14:48:35.840782 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:35.840563    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b076be2_0dd9_4f62_bef3_e00ecb467b25.slice/crio-1e4ced30b0139be70da516415001938d5840664b660aafc8dd695d4e5612ac25 WatchSource:0}: Error finding container 1e4ced30b0139be70da516415001938d5840664b660aafc8dd695d4e5612ac25: Status 404 returned error can't find the container with id 1e4ced30b0139be70da516415001938d5840664b660aafc8dd695d4e5612ac25
Jul 31 14:48:35.840935 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.840895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:35.955758 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:35.955675    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpxk6_50c437ff-669d-4c9f-aa01-039af0c80380/node-ca/0.log"
Jul 31 14:48:36.260062 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.259971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:36.260224 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.260076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:36.260224 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:36.260157    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:36.260224 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:36.260217    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:36.260334 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:36.260230    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdf7c5b-2wgc5: secret "image-registry-tls" not found
Jul 31 14:48:36.260334 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:36.260239    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls podName:558d0e42-9039-431d-b66d-51770b2ef959 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:40.260214028 +0000 UTC m=+43.594333942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls") pod "dns-default-czhw9" (UID: "558d0e42-9039-431d-b66d-51770b2ef959") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:36.260334 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:36.260269    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls podName:87744021-57ed-452d-8187-173d2f849f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:40.260256349 +0000 UTC m=+43.594376249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls") pod "image-registry-55fdf7c5b-2wgc5" (UID: "87744021-57ed-452d-8187-173d2f849f42") : secret "image-registry-tls" not found
Jul 31 14:48:36.459609 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.459566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn" event={"ID":"3b076be2-0dd9-4f62-bef3-e00ecb467b25","Type":"ContainerStarted","Data":"1e4ced30b0139be70da516415001938d5840664b660aafc8dd695d4e5612ac25"}
Jul 31 14:48:36.463303 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.463269    2575 generic.go:356] "Generic (PLEG): container finished" podID="56403f34-c078-4c81-805c-7882ec9e5284" containerID="b43a105958be0ccdecc9d569a5a92f81adbf68223259e77cad3da141720aad98" exitCode=0
Jul 31 14:48:36.463462 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.463318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerDied","Data":"b43a105958be0ccdecc9d569a5a92f81adbf68223259e77cad3da141720aad98"}
Jul 31 14:48:36.725013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.724974    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:36.793886 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:36.793852    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:37.157768 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.157724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:37.287405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.287368    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:37.435394 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.435365    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:37.471042 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.471010    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:37.471192 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.471069    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:37.471192 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:37.471159    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:37.471192 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:37.471169    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:37.471286 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:37.471213    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.4711983 +0000 UTC m=+44.805318200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:37.471286 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:37.471226    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.471219472 +0000 UTC m=+44.805339371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:37.873166 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.873101    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:37.990151 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:37.990110    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:38.425644 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:38.425575    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:38.932942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:38.932908    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:39.035270 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.035231    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:39.475139 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.475060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5shsp" event={"ID":"56403f34-c078-4c81-805c-7882ec9e5284","Type":"ContainerStarted","Data":"0349fb37796d037c10cdeb80f99ff825378f283ff979a2fdacb32c42083ef0dd"}
Jul 31 14:48:39.477887 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.477824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s" event={"ID":"a97a064f-de87-4f33-9387-5384c5967a2e","Type":"ContainerStarted","Data":"5e6bb2dadc020be3d2cd1d50e23432b60915b94e9840a92ba55103526d25b689"}
Jul 31 14:48:39.479751 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.479701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zx94w" event={"ID":"22d806cf-8c2e-4a36-a22f-f7e32bb1f619","Type":"ContainerStarted","Data":"a13eef6d838473c70e9ba2f9f7a877450385d6b4e0295bb88728dd4dc3ee98f5"}
Jul 31 14:48:39.481634 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.481584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn" event={"ID":"3b076be2-0dd9-4f62-bef3-e00ecb467b25","Type":"ContainerStarted","Data":"d3463f913ed44c720ff98637e990ac2633d719f5ff091e732df792aa8bd37264"}
Jul 31 14:48:39.517592 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.517538    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5shsp" podStartSLOduration=10.404962284 podStartE2EDuration="42.517522226s" totalImagesPullingTime="32.112559942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:39.516440131 +0000 UTC m=+42.850560054" watchObservedRunningTime="2026-07-31 14:48:39.517522226 +0000 UTC m=+42.851642147"
Jul 31 14:48:39.567127 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.567042    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-d724s" podStartSLOduration=2.073145902 podStartE2EDuration="6.567024105s" totalImagesPullingTime="4.493878203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:39.566290425 +0000 UTC m=+42.900410349" watchObservedRunningTime="2026-07-31 14:48:39.567024105 +0000 UTC m=+42.901144028"
Jul 31 14:48:39.567370 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:39.567286    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zx94w" podStartSLOduration=3.084036993 podStartE2EDuration="7.567278503s" totalImagesPullingTime="4.48324151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:39.544350423 +0000 UTC m=+42.878470345" watchObservedRunningTime="2026-07-31 14:48:39.567278503 +0000 UTC m=+42.901398425"
Jul 31 14:48:40.296663 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:40.296621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:40.296689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:40.296771    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:40.296819    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:40.296833    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55fdf7c5b-2wgc5: secret "image-registry-tls" not found
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:40.296850    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls podName:558d0e42-9039-431d-b66d-51770b2ef959 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:48.296832539 +0000 UTC m=+51.630952442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls") pod "dns-default-czhw9" (UID: "558d0e42-9039-431d-b66d-51770b2ef959") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:40.297049 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:40.296871    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls podName:87744021-57ed-452d-8187-173d2f849f42 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:48.296860432 +0000 UTC m=+51.630980332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls") pod "image-registry-55fdf7c5b-2wgc5" (UID: "87744021-57ed-452d-8187-173d2f849f42") : secret "image-registry-tls" not found
Jul 31 14:48:40.487983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:40.487947    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn" event={"ID":"3b076be2-0dd9-4f62-bef3-e00ecb467b25","Type":"ContainerStarted","Data":"4a05f8cdd4c96129f531766c33ac7fea474846e066c46a8df12d0c58af2e8fb6"}
Jul 31 14:48:40.510676 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:40.510619    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-d4jdn" podStartSLOduration=2.1577647620000002 podStartE2EDuration="5.510581296s" totalImagesPullingTime="3.352816534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:40.510040236 +0000 UTC m=+43.844160158" watchObservedRunningTime="2026-07-31 14:48:40.510581296 +0000 UTC m=+43.844701231"
Jul 31 14:48:41.111611 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.111566    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:41.360100 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.360062    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:41.462292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.462190    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:41.505948 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.505911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:41.506173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.506036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:41.506173 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:41.506130    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:41.506302 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:41.506179    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:41.506302 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:41.506198    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:49.506180945 +0000 UTC m=+52.840300845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:41.506302 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:41.506230    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:49.506215513 +0000 UTC m=+52.840335413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:41.772565 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.772428    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:41.868456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:41.868425    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:42.455879 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:42.455829    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:42.538780 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:42.538747    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:43.246952 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:43.246894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:43.630266 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:43.630203    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:43.854251 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:43.854187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:44.058686 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:44.058512    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:44.412340 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:44.412306    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:44.814024 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:44.813899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:45.037350 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:45.037284    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:45.428126 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:45.428097    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:45.510255 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:45.510223    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:45.814102 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:45.814019    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:46.217836 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:46.217798    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:46.322621 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:46.322570    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:46.436621 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:46.436572    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:46.562789 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:46.562702    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:47.013876 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:47.013846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:47.227638 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:47.227588    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:47.821387 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:47.821355    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:48.203389 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.203348    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:48.367754 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.367717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:48.367947 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.367787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:48.371240 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.371206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"image-registry-55fdf7c5b-2wgc5\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") " pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:48.371372 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.371208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/558d0e42-9039-431d-b66d-51770b2ef959-metrics-tls\") pod \"dns-default-czhw9\" (UID: \"558d0e42-9039-431d-b66d-51770b2ef959\") " pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:48.448728 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.448695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:48.467887 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.467818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:48.650857 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.650813    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:48:48.656549 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:48.656508    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87744021_57ed_452d_8187_173d2f849f42.slice/crio-e62bfe34c4b83fa7236777f0812030ed7787e56eb4684dbf34db6deca75059e2 WatchSource:0}: Error finding container e62bfe34c4b83fa7236777f0812030ed7787e56eb4684dbf34db6deca75059e2: Status 404 returned error can't find the container with id e62bfe34c4b83fa7236777f0812030ed7787e56eb4684dbf34db6deca75059e2
Jul 31 14:48:48.677134 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:48.677098    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558d0e42_9039_431d_b66d_51770b2ef959.slice/crio-2079e3e225c8180659187b38a4ed61a034c840c105db3f6ebc9b68e8d56c3b20 WatchSource:0}: Error finding container 2079e3e225c8180659187b38a4ed61a034c840c105db3f6ebc9b68e8d56c3b20: Status 404 returned error can't find the container with id 2079e3e225c8180659187b38a4ed61a034c840c105db3f6ebc9b68e8d56c3b20
Jul 31 14:48:48.685314 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:48.685285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-czhw9"]
Jul 31 14:48:49.178205 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.178167    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:49.304496 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.304446    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:49.512759 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.512666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czhw9" event={"ID":"558d0e42-9039-431d-b66d-51770b2ef959","Type":"ContainerStarted","Data":"2079e3e225c8180659187b38a4ed61a034c840c105db3f6ebc9b68e8d56c3b20"}
Jul 31 14:48:49.514300 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.514268    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" event={"ID":"87744021-57ed-452d-8187-173d2f849f42","Type":"ContainerStarted","Data":"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"}
Jul 31 14:48:49.514448 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.514316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" event={"ID":"87744021-57ed-452d-8187-173d2f849f42","Type":"ContainerStarted","Data":"e62bfe34c4b83fa7236777f0812030ed7787e56eb4684dbf34db6deca75059e2"}
Jul 31 14:48:49.514448 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.514351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:48:49.551769 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.551713    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" podStartSLOduration=21.551688294 podStartE2EDuration="21.551688294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:49.55109577 +0000 UTC m=+52.885215693" watchObservedRunningTime="2026-07-31 14:48:49.551688294 +0000 UTC m=+52.885808218"
Jul 31 14:48:49.578073 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.578038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:49.578280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.578119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:49.578280 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:49.578209    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:49.578401 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:48:49.578285    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls podName:ebf922ca-e856-43c9-8e88-899512ec8fbe nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.57826677 +0000 UTC m=+68.912386673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2w2zm" (UID: "ebf922ca-e856-43c9-8e88-899512ec8fbe") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:49.578825 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.578797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ebf922ca-e856-43c9-8e88-899512ec8fbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:48:49.632768 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:49.632710    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:50.248576 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:50.248532    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:50.449232 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:50.449198    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:50.606763 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:50.606730    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:50.844189 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:50.844159    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:50.947199 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:50.947113    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:51.047132 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.047088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:51.211766 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.211728    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:51.405586 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.405552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:51.523565 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.523468    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czhw9" event={"ID":"558d0e42-9039-431d-b66d-51770b2ef959","Type":"ContainerStarted","Data":"0e29e1f8320b7060aa42621f4f702c2e3132b4c717e38055098c259bd32473ce"}
Jul 31 14:48:51.523565 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.523535    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-czhw9" event={"ID":"558d0e42-9039-431d-b66d-51770b2ef959","Type":"ContainerStarted","Data":"4d08db5f97c338691f1c5a524433984e3b4b59684d2a4ce7565d2dcc82fc0865"}
Jul 31 14:48:51.524028 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.523578    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-czhw9"
Jul 31 14:48:51.545208 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.545153    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-czhw9" podStartSLOduration=17.235051734 podStartE2EDuration="19.545136864s" totalImagesPullingTime="2.31008513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:51.543660769 +0000 UTC m=+54.877780691" watchObservedRunningTime="2026-07-31 14:48:51.545136864 +0000 UTC m=+54.879256785"
Jul 31 14:48:51.801828 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:51.801741    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:57.906803 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.906771    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"]
Jul 31 14:48:57.932771 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.932743    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"]
Jul 31 14:48:57.932937 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.932833    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:57.937696 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.937670    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 14:48:57.938866 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.938838    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xf94s\""
Jul 31 14:48:57.939027 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.938886    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 14:48:57.939173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.939136    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 14:48:57.939239 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.939209    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 14:48:57.942764 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.942695    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2ch\" (UniqueName: \"kubernetes.io/projected/939dbf91-e107-4736-b867-bd0cff420666-kube-api-access-2b2ch\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:57.942898 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.942830    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939dbf91-e107-4736-b867-bd0cff420666-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:57.945188 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:57.945165    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:48:58.030456 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.030420    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:58.043378 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.043341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939dbf91-e107-4736-b867-bd0cff420666-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:58.043585 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.043405    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2ch\" (UniqueName: \"kubernetes.io/projected/939dbf91-e107-4736-b867-bd0cff420666-kube-api-access-2b2ch\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:58.046280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.046253    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/939dbf91-e107-4736-b867-bd0cff420666-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:58.059687 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.059654    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-679dfb877-kg964"]
Jul 31 14:48:58.061637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.061619    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.077930 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.077900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2ch\" (UniqueName: \"kubernetes.io/projected/939dbf91-e107-4736-b867-bd0cff420666-kube-api-access-2b2ch\") pod \"managed-serviceaccount-addon-agent-6666d6f488-99flc\" (UID: \"939dbf91-e107-4736-b867-bd0cff420666\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:58.091473 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.091443    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-679dfb877-kg964"]
Jul 31 14:48:58.144117 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-tls\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144117 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-ca-trust-extracted\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144365 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-image-registry-private-configuration\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144365 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-installation-pull-secrets\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144365 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-trusted-ca\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144365 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mnr\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-kube-api-access-c7mnr\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-certificates\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.144513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.144450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-bound-sa-token\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.149520 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.149488    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:58.245788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-tls\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.245788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-ca-trust-extracted\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.245788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-image-registry-private-configuration\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.245788 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-installation-pull-secrets\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246121 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-trusted-ca\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246121 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.245995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mnr\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-kube-api-access-c7mnr\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246121 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.246058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-certificates\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246121 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.246116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-bound-sa-token\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.246237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-ca-trust-extracted\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246808 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.246784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-certificates\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.246808 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.246797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-trusted-ca\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.248662 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.248641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-registry-tls\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.249025 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.249001    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-installation-pull-secrets\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.249064 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.249027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-image-registry-private-configuration\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.251872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.251848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"
Jul 31 14:48:58.276829 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.276797    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mnr\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-kube-api-access-c7mnr\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.286316 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.286285    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/291bd47a-9d3f-461f-8a23-64c5cb07ebbd-bound-sa-token\") pod \"image-registry-679dfb877-kg964\" (UID: \"291bd47a-9d3f-461f-8a23-64c5cb07ebbd\") " pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.370977 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.370942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:58.390004 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.389970    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc"]
Jul 31 14:48:58.393988 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:58.393938    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod939dbf91_e107_4736_b867_bd0cff420666.slice/crio-c9ac2de38e936f983be54800c9e75a8f99dae9a2a3339b91cbaa4bc0d1f2a4d8 WatchSource:0}: Error finding container c9ac2de38e936f983be54800c9e75a8f99dae9a2a3339b91cbaa4bc0d1f2a4d8: Status 404 returned error can't find the container with id c9ac2de38e936f983be54800c9e75a8f99dae9a2a3339b91cbaa4bc0d1f2a4d8
Jul 31 14:48:58.438375 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.438347    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5w4ll"
Jul 31 14:48:58.517223 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.517194    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-679dfb877-kg964"]
Jul 31 14:48:58.517365 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:48:58.517248    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291bd47a_9d3f_461f_8a23_64c5cb07ebbd.slice/crio-c63f018ee7013dcd1161226e2e38b551d1d568b14c485f56013bcf0a6d006304 WatchSource:0}: Error finding container c63f018ee7013dcd1161226e2e38b551d1d568b14c485f56013bcf0a6d006304: Status 404 returned error can't find the container with id c63f018ee7013dcd1161226e2e38b551d1d568b14c485f56013bcf0a6d006304
Jul 31 14:48:58.549224 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.549189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc" event={"ID":"939dbf91-e107-4736-b867-bd0cff420666","Type":"ContainerStarted","Data":"c9ac2de38e936f983be54800c9e75a8f99dae9a2a3339b91cbaa4bc0d1f2a4d8"}
Jul 31 14:48:58.550416 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.550390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-679dfb877-kg964" event={"ID":"291bd47a-9d3f-461f-8a23-64c5cb07ebbd","Type":"ContainerStarted","Data":"c63f018ee7013dcd1161226e2e38b551d1d568b14c485f56013bcf0a6d006304"}
Jul 31 14:48:58.670638 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.670584    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:58.769348 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:58.769259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:59.059139 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:59.059051    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:59.174463 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:59.174403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:48:59.560747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:59.560701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-679dfb877-kg964" event={"ID":"291bd47a-9d3f-461f-8a23-64c5cb07ebbd","Type":"ContainerStarted","Data":"5e46f6b589a74ca63440ab33f532af139300266abf836f5cdf73191f9c2c7f32"}
Jul 31 14:48:59.561019 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:59.560999    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:48:59.590209 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:48:59.590146    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-679dfb877-kg964" podStartSLOduration=1.590123365 podStartE2EDuration="1.590123365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:59.588336178 +0000 UTC m=+62.922456100" watchObservedRunningTime="2026-07-31 14:48:59.590123365 +0000 UTC m=+62.924243282"
Jul 31 14:49:00.131479 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:00.131440    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:00.241646 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:00.241592    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:00.354518 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:00.354473    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:01.530517 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:01.530484    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-czhw9"
Jul 31 14:49:01.569944 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:01.569907    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc" event={"ID":"939dbf91-e107-4736-b867-bd0cff420666","Type":"ContainerStarted","Data":"af488c4498baa4d4ffb169ae0ffcd014412fa347ebcc7ad14380540bb2bd2eb9"}
Jul 31 14:49:01.587862 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:01.587812    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6666d6f488-99flc" podStartSLOduration=1.9725892919999999 podStartE2EDuration="4.587795527s" totalImagesPullingTime="2.615206235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:01.586684835 +0000 UTC m=+64.920804753" watchObservedRunningTime="2026-07-31 14:49:01.587795527 +0000 UTC m=+64.921915448"
Jul 31 14:49:02.460020 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.459984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:02.575619 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.575564    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:02.683444 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.683409    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"]
Jul 31 14:49:02.685413 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.685389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:02.688718 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.688692    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hzd6c\""
Jul 31 14:49:02.689730 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.689713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 31 14:49:02.700066 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.700036    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"]
Jul 31 14:49:02.748999 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.748912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:02.790206 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.790174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-srsgc\" (UID: \"ce21b351-9a55-4290-92e2-d5836715aa40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:02.890897 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.890862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-srsgc\" (UID: \"ce21b351-9a55-4290-92e2-d5836715aa40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:02.891027 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:02.891012    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:02.891093 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:02.891082    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates podName:ce21b351-9a55-4290-92e2-d5836715aa40 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.391064134 +0000 UTC m=+66.725184040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-srsgc" (UID: "ce21b351-9a55-4290-92e2-d5836715aa40") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:02.991962 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.991930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:49:02.992152 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.991999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:49:02.995253 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.995228    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:02.995382 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:02.995228    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:03.005264 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.005188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94357ff7-eec1-4ce0-b983-f574a7204766-original-pull-secret\") pod \"global-pull-secret-syncer-nt9j7\" (UID: \"94357ff7-eec1-4ce0-b983-f574a7204766\") " pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:49:03.005264 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.005199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61802f40-474e-4d7d-94ef-a3bfc53e6a6f-metrics-certs\") pod \"network-metrics-daemon-xcfwb\" (UID: \"61802f40-474e-4d7d-94ef-a3bfc53e6a6f\") " pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:49:03.092770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.092715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:49:03.095414 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.095382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgcc\" (UniqueName: \"kubernetes.io/projected/0465a7ba-0b2d-4366-bbdf-27bacfa77a2c-kube-api-access-4zgcc\") pod \"network-check-target-dt4ln\" (UID: \"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c\") " pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:49:03.208060 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.208018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nt9j7"
Jul 31 14:49:03.218162 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.218133    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m4v8n\""
Jul 31 14:49:03.224764 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.224732    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2kc8\""
Jul 31 14:49:03.225675 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.225657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xcfwb"
Jul 31 14:49:03.232779 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.232742    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:49:03.373704 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.372586    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nt9j7"]
Jul 31 14:49:03.377922 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:03.377879    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94357ff7_eec1_4ce0_b983_f574a7204766.slice/crio-ad48d323a2df18563e67e0f5e0f8be815212b6cfa635fa0b64bec19b1303a7c9 WatchSource:0}: Error finding container ad48d323a2df18563e67e0f5e0f8be815212b6cfa635fa0b64bec19b1303a7c9: Status 404 returned error can't find the container with id ad48d323a2df18563e67e0f5e0f8be815212b6cfa635fa0b64bec19b1303a7c9
Jul 31 14:49:03.394993 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.394962    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dt4ln"]
Jul 31 14:49:03.395321 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.395291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-srsgc\" (UID: \"ce21b351-9a55-4290-92e2-d5836715aa40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:03.395404 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:03.395394    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:03.395475 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:03.395464    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates podName:ce21b351-9a55-4290-92e2-d5836715aa40 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:04.395446563 +0000 UTC m=+67.729566463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-srsgc" (UID: "ce21b351-9a55-4290-92e2-d5836715aa40") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:03.398868 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:03.398841    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0465a7ba_0b2d_4366_bbdf_27bacfa77a2c.slice/crio-74d29c9a3b9f7640a860a2e9d5b6b95bd0f68c1de52aa3bbaa97da11939fbfbe WatchSource:0}: Error finding container 74d29c9a3b9f7640a860a2e9d5b6b95bd0f68c1de52aa3bbaa97da11939fbfbe: Status 404 returned error can't find the container with id 74d29c9a3b9f7640a860a2e9d5b6b95bd0f68c1de52aa3bbaa97da11939fbfbe
Jul 31 14:49:03.412179 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.412150    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xcfwb"]
Jul 31 14:49:03.415860 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:03.415835    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61802f40_474e_4d7d_94ef_a3bfc53e6a6f.slice/crio-478bbc0840ebe6bc3c7b4228a7922e9b5a142c0178a96e7d3a6a6c3c16f931b5 WatchSource:0}: Error finding container 478bbc0840ebe6bc3c7b4228a7922e9b5a142c0178a96e7d3a6a6c3c16f931b5: Status 404 returned error can't find the container with id 478bbc0840ebe6bc3c7b4228a7922e9b5a142c0178a96e7d3a6a6c3c16f931b5
Jul 31 14:49:03.487556 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.487510    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:03.578496 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.578400    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nt9j7" event={"ID":"94357ff7-eec1-4ce0-b983-f574a7204766","Type":"ContainerStarted","Data":"ad48d323a2df18563e67e0f5e0f8be815212b6cfa635fa0b64bec19b1303a7c9"}
Jul 31 14:49:03.580158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.580129    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dt4ln" event={"ID":"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c","Type":"ContainerStarted","Data":"997d0a619d1e9d78c7df7f96a8b1e24d7f1d799f3e092961497d0fc237927522"}
Jul 31 14:49:03.580301 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.580171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dt4ln" event={"ID":"0465a7ba-0b2d-4366-bbdf-27bacfa77a2c","Type":"ContainerStarted","Data":"74d29c9a3b9f7640a860a2e9d5b6b95bd0f68c1de52aa3bbaa97da11939fbfbe"}
Jul 31 14:49:03.580301 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.580237    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:49:03.581419 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.581393    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcfwb" event={"ID":"61802f40-474e-4d7d-94ef-a3bfc53e6a6f","Type":"ContainerStarted","Data":"478bbc0840ebe6bc3c7b4228a7922e9b5a142c0178a96e7d3a6a6c3c16f931b5"}
Jul 31 14:49:03.604126 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:03.604052    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dt4ln" podStartSLOduration=65.604031885 podStartE2EDuration="1m5.604031885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:03.603818607 +0000 UTC m=+66.937938530" watchObservedRunningTime="2026-07-31 14:49:03.604031885 +0000 UTC m=+66.938151787"
Jul 31 14:49:04.402969 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:04.402930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-srsgc\" (UID: \"ce21b351-9a55-4290-92e2-d5836715aa40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:04.406242 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:04.406203    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ce21b351-9a55-4290-92e2-d5836715aa40-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-srsgc\" (UID: \"ce21b351-9a55-4290-92e2-d5836715aa40\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:04.494791 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:04.494739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:04.718442 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:04.718315    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"]
Jul 31 14:49:04.721071 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:04.721029    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce21b351_9a55_4290_92e2_d5836715aa40.slice/crio-1f12d5819e92d421b510726668f20cd76cc55c09755ae6e5c1b7b5150fb5d326 WatchSource:0}: Error finding container 1f12d5819e92d421b510726668f20cd76cc55c09755ae6e5c1b7b5150fb5d326: Status 404 returned error can't find the container with id 1f12d5819e92d421b510726668f20cd76cc55c09755ae6e5c1b7b5150fb5d326
Jul 31 14:49:05.335932 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.335868    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:05.429021 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.428992    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:05.594329 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.594232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc" event={"ID":"ce21b351-9a55-4290-92e2-d5836715aa40","Type":"ContainerStarted","Data":"1f12d5819e92d421b510726668f20cd76cc55c09755ae6e5c1b7b5150fb5d326"}
Jul 31 14:49:05.596361 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.596327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcfwb" event={"ID":"61802f40-474e-4d7d-94ef-a3bfc53e6a6f","Type":"ContainerStarted","Data":"326588baf3982b3823e2d5c8e897783a7e7acbeb50dbd184cdeb573464aa53af"}
Jul 31 14:49:05.596492 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.596386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xcfwb" event={"ID":"61802f40-474e-4d7d-94ef-a3bfc53e6a6f","Type":"ContainerStarted","Data":"c57399a00231875f3fdbef4660ce3677c0937c4852ec7b9656b852ecdd9f342f"}
Jul 31 14:49:05.612144 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.612103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:49:05.615153 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.615113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ebf922ca-e856-43c9-8e88-899512ec8fbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2w2zm\" (UID: \"ebf922ca-e856-43c9-8e88-899512ec8fbe\") " pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:49:05.619138 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.619078    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xcfwb" podStartSLOduration=67.113040339 podStartE2EDuration="1m8.619057242s" totalImagesPullingTime="1.506016903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:05.617160426 +0000 UTC m=+68.951280351" watchObservedRunningTime="2026-07-31 14:49:05.619057242 +0000 UTC m=+68.953177165"
Jul 31 14:49:05.758973 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.758937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:05.812471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.812439    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gfnvf\""
Jul 31 14:49:05.816873 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.816832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2w2zm"
Jul 31 14:49:05.871807 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:05.871771    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:06.013355 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:06.013318    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2w2zm"]
Jul 31 14:49:06.602757 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:06.602719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc" event={"ID":"ce21b351-9a55-4290-92e2-d5836715aa40","Type":"ContainerStarted","Data":"8b78b88f46a4f69d0d02a705125c1122d417116cf44acb81e49dee1a53e7552d"}
Jul 31 14:49:06.624011 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:06.623949    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc" podStartSLOduration=3.462643547 podStartE2EDuration="4.623930131s" totalImagesPullingTime="1.161286584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:06.623069771 +0000 UTC m=+69.957189694" watchObservedRunningTime="2026-07-31 14:49:06.623930131 +0000 UTC m=+69.958050054"
Jul 31 14:49:07.312403 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:07.312366    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf922ca_e856_43c9_8e88_899512ec8fbe.slice/crio-fc47d881e8ee62709dacc00acfe048b5f86de91c73a38cd863fa8f8daba75339 WatchSource:0}: Error finding container fc47d881e8ee62709dacc00acfe048b5f86de91c73a38cd863fa8f8daba75339: Status 404 returned error can't find the container with id fc47d881e8ee62709dacc00acfe048b5f86de91c73a38cd863fa8f8daba75339
Jul 31 14:49:07.608026 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.607986    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nt9j7" event={"ID":"94357ff7-eec1-4ce0-b983-f574a7204766","Type":"ContainerStarted","Data":"8daefd9f38e0286a9778e54b7831adfcf095d320e585b4755c36f766a5d1a58c"}
Jul 31 14:49:07.609664 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.609634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w2zm" event={"ID":"ebf922ca-e856-43c9-8e88-899512ec8fbe","Type":"ContainerStarted","Data":"f4fafea5d7146c9246a1c12657479ff0543c8e1701e2f0d70cbbc317cff23e86"}
Jul 31 14:49:07.609822 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.609691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w2zm" event={"ID":"ebf922ca-e856-43c9-8e88-899512ec8fbe","Type":"ContainerStarted","Data":"fc47d881e8ee62709dacc00acfe048b5f86de91c73a38cd863fa8f8daba75339"}
Jul 31 14:49:07.609876 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.609816    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:07.614840 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.614812    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-srsgc"
Jul 31 14:49:07.627953 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.627897    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nt9j7" podStartSLOduration=65.659482077 podStartE2EDuration="1m9.627878875s" totalImagesPullingTime="3.968396798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:07.627193082 +0000 UTC m=+70.961313004" watchObservedRunningTime="2026-07-31 14:49:07.627878875 +0000 UTC m=+70.961998797"
Jul 31 14:49:07.950903 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.950866    2575 patch_prober.go:28] interesting pod/image-registry-55fdf7c5b-2wgc5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 14:49:07.951037 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:07.950925    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" podUID="87744021-57ed-452d-8187-173d2f849f42" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:49:08.249336 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.249241    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:08.327926 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.327890    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:08.616392 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.616356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w2zm" event={"ID":"ebf922ca-e856-43c9-8e88-899512ec8fbe","Type":"ContainerStarted","Data":"27fe9ab06655fd73ccf1062cfd0dad49b9faf5c8ba09eeb6ef7fc672bede24ba"}
Jul 31 14:49:08.708506 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.708471    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:08.768048 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.768007    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"]
Jul 31 14:49:08.771784 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.771746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.775452 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.775416    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 31 14:49:08.776316 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.776114    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t7g2l\""
Jul 31 14:49:08.776785 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.776760    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 31 14:49:08.776904 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.776802    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:49:08.777262 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.777228    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:49:08.777262 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.777246    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:08.799329 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.784300    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"]
Jul 31 14:49:08.810160 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.810123    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:08.837320 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.837278    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5md\" (UniqueName: \"kubernetes.io/projected/e8311c6e-336c-4254-89e7-e4bee8e6c877-kube-api-access-rq5md\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.837503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.837408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.837503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.837445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8311c6e-336c-4254-89e7-e4bee8e6c877-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.837503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.837477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.938281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.938188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.938281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.938237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8311c6e-336c-4254-89e7-e4bee8e6c877-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.938529 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.938364    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.938529 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.938411    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5md\" (UniqueName: \"kubernetes.io/projected/e8311c6e-336c-4254-89e7-e4bee8e6c877-kube-api-access-rq5md\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.938668 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:08.938534    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 31 14:49:08.938668 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:08.938637    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls podName:e8311c6e-336c-4254-89e7-e4bee8e6c877 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.438587236 +0000 UTC m=+72.772707156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-l9hxs" (UID: "e8311c6e-336c-4254-89e7-e4bee8e6c877") : secret "prometheus-operator-tls" not found
Jul 31 14:49:08.939287 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.939265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8311c6e-336c-4254-89e7-e4bee8e6c877-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.941178 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.941155    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:08.962013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:08.961980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5md\" (UniqueName: \"kubernetes.io/projected/e8311c6e-336c-4254-89e7-e4bee8e6c877-kube-api-access-rq5md\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:09.072737 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.072688    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:09.442184 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.442139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:09.444739 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.444714    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/e8311c6e-336c-4254-89e7-e4bee8e6c877-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-l9hxs\" (UID: \"e8311c6e-336c-4254-89e7-e4bee8e6c877\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:09.622301 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.622253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2w2zm" event={"ID":"ebf922ca-e856-43c9-8e88-899512ec8fbe","Type":"ContainerStarted","Data":"d8f67aaaf85a01a0ef1aef05cc7038ed60a7a11cb687ca0b21b6fb2602dbf013"}
Jul 31 14:49:09.643620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.643550    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2w2zm" podStartSLOduration=34.942600035 podStartE2EDuration="36.643533615s" totalImagesPullingTime="1.70093358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:09.642338452 +0000 UTC m=+72.976458375" watchObservedRunningTime="2026-07-31 14:49:09.643533615 +0000 UTC m=+72.977653732"
Jul 31 14:49:09.683365 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.683331    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"
Jul 31 14:49:09.827218 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:09.827183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-l9hxs"]
Jul 31 14:49:09.830652 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:09.830622    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8311c6e_336c_4254_89e7_e4bee8e6c877.slice/crio-92f07cc866bf247c9a0e4494b8c4de19e042089304eb1970a9e8d5935374ff35 WatchSource:0}: Error finding container 92f07cc866bf247c9a0e4494b8c4de19e042089304eb1970a9e8d5935374ff35: Status 404 returned error can't find the container with id 92f07cc866bf247c9a0e4494b8c4de19e042089304eb1970a9e8d5935374ff35
Jul 31 14:49:10.626669 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:10.626631    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs" event={"ID":"e8311c6e-336c-4254-89e7-e4bee8e6c877","Type":"ContainerStarted","Data":"92f07cc866bf247c9a0e4494b8c4de19e042089304eb1970a9e8d5935374ff35"}
Jul 31 14:49:10.689671 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:10.689638    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:10.787563 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:10.787526    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:11.928218 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:11.928181    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:12.025022 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.024914    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:12.091039 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.091002    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:12.635707 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.635671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs" event={"ID":"e8311c6e-336c-4254-89e7-e4bee8e6c877","Type":"ContainerStarted","Data":"9cf8838746938ff6254284f52cea2b8ff5f9d09a795e7809f89d52abf0dfb64b"}
Jul 31 14:49:12.635894 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.635724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs" event={"ID":"e8311c6e-336c-4254-89e7-e4bee8e6c877","Type":"ContainerStarted","Data":"b66c1ec816162ce55a75a44ccb1702965379099b7a89ba8acd82e98e3e6a8aff"}
Jul 31 14:49:12.657907 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.657852    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-l9hxs" podStartSLOduration=2.803733514 podStartE2EDuration="4.657832601s" totalImagesPullingTime="1.854099087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:12.656726406 +0000 UTC m=+75.990846327" watchObservedRunningTime="2026-07-31 14:49:12.657832601 +0000 UTC m=+75.991952523"
Jul 31 14:49:12.774332 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.774302    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:12.875326 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.875292    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:12.950056 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:12.949979    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:13.068916 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:13.068882    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:13.455393 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:13.455358    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:13.654635 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:13.654581    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:13.720359 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:13.720279    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:13.890883 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:13.890842    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.164898 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.164864    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.374013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.373984    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.517152 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.517068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.574654 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.574620    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cf5wf"]
Jul 31 14:49:14.577317 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.577296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.579581 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.579558    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:49:14.580144 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.580124    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:49:14.580251 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.580181    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:49:14.581233 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.581218    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qmq89\""
Jul 31 14:49:14.632790 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.632762    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.680886 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.680843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-wtmp\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.680886 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.680884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-root\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.680905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-metrics-client-ca\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vq7\" (UniqueName: \"kubernetes.io/projected/875788cb-55ed-4a00-a657-4a418eb96d63-kube-api-access-b6vq7\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-sys\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-tls\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-textfile\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681300 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-accelerators-collector-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.681300 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.681185    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.745240 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.745206    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.781642 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-wtmp\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781642 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-root\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781670    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-root\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781694    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-metrics-client-ca\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vq7\" (UniqueName: \"kubernetes.io/projected/875788cb-55ed-4a00-a657-4a418eb96d63-kube-api-access-b6vq7\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-wtmp\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-sys\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-tls\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.781818 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-textfile\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782107 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-accelerators-collector-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782107 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782107 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.781860    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/875788cb-55ed-4a00-a657-4a418eb96d63-sys\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782254 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.782176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-textfile\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782332 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.782312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-metrics-client-ca\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.782467 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.782447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-accelerators-collector-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.784197 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.784176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.784358 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.784340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/875788cb-55ed-4a00-a657-4a418eb96d63-node-exporter-tls\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.790802 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.790780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vq7\" (UniqueName: \"kubernetes.io/projected/875788cb-55ed-4a00-a657-4a418eb96d63-kube-api-access-b6vq7\") pod \"node-exporter-cf5wf\" (UID: \"875788cb-55ed-4a00-a657-4a418eb96d63\") " pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.845409 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.845347    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:14.886654 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.886591    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cf5wf"
Jul 31 14:49:14.897323 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:14.897289    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875788cb_55ed_4a00_a657_4a418eb96d63.slice/crio-3f37e1a9b8e5111e9243960d891d8bbd31c2929d9743fdf8cff1fc674f6dcc2a WatchSource:0}: Error finding container 3f37e1a9b8e5111e9243960d891d8bbd31c2929d9743fdf8cff1fc674f6dcc2a: Status 404 returned error can't find the container with id 3f37e1a9b8e5111e9243960d891d8bbd31c2929d9743fdf8cff1fc674f6dcc2a
Jul 31 14:49:14.944549 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:14.944514    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:15.046227 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.046136    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:15.206256 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.206220    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:15.318506 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.318407    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:15.501953 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.501918    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:15.648011 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.647971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cf5wf" event={"ID":"875788cb-55ed-4a00-a657-4a418eb96d63","Type":"ContainerStarted","Data":"3f37e1a9b8e5111e9243960d891d8bbd31c2929d9743fdf8cff1fc674f6dcc2a"}
Jul 31 14:49:15.834211 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:15.834169    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:16.195391 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:16.195300    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:16.316986 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:16.316956    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:16.653247 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:16.653215    2575 generic.go:356] "Generic (PLEG): container finished" podID="875788cb-55ed-4a00-a657-4a418eb96d63" containerID="d7082572c9907aecbf054979918d6b225aedb5a7eae9aa92cda557f10978e170" exitCode=0
Jul 31 14:49:16.653434 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:16.653278    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cf5wf" event={"ID":"875788cb-55ed-4a00-a657-4a418eb96d63","Type":"ContainerDied","Data":"d7082572c9907aecbf054979918d6b225aedb5a7eae9aa92cda557f10978e170"}
Jul 31 14:49:17.092048 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.091958    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:17.217215 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.217176    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:17.420522 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.420484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:17.432798 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.432768    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"]
Jul 31 14:49:17.435661 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.435642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.439184 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.439161    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 14:49:17.439184 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.439163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rdpqg\""
Jul 31 14:49:17.439349 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.439194    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 14:49:17.439731 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.439717    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 14:49:17.440163 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.440145    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dhiijkivrki78\""
Jul 31 14:49:17.440247 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.440165    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 14:49:17.440247 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.440173    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 14:49:17.466177 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.466138    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"]
Jul 31 14:49:17.505050 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505235 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505235 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6z7\" (UniqueName: \"kubernetes.io/projected/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-kube-api-access-hj6z7\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505235 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505235 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505202    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505235 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505222    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505419 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-metrics-client-ca\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.505419 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.505255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-grpc-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.551988 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.551952    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:17.606445 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606404    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6z7\" (UniqueName: \"kubernetes.io/projected/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-kube-api-access-hj6z7\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606517    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606536    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606673 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606551    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-metrics-client-ca\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606930 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-grpc-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.606930 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.606790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.607619 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.607567    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-metrics-client-ca\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.609505 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.609472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.609638 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.609544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-grpc-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.609757 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.609737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.609901 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.609877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.609978 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.609956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-tls\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.610284 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.610263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.621522 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.621489    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6z7\" (UniqueName: \"kubernetes.io/projected/410bfc4b-b27a-4740-b7a9-ecfd14529b0b-kube-api-access-hj6z7\") pod \"thanos-querier-7fb9c6997f-vknzq\" (UID: \"410bfc4b-b27a-4740-b7a9-ecfd14529b0b\") " pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.659464 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.659419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cf5wf" event={"ID":"875788cb-55ed-4a00-a657-4a418eb96d63","Type":"ContainerStarted","Data":"29efad461b811f8c91c0d2d9d1c21ed65fa5e9fe7fd070524ea2e9d8b6adbcb9"}
Jul 31 14:49:17.659464 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.659475    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cf5wf" event={"ID":"875788cb-55ed-4a00-a657-4a418eb96d63","Type":"ContainerStarted","Data":"72398f23696763e28ca1ee6908b9fe4c43200f0f6702b27cf5349d636e4da6b5"}
Jul 31 14:49:17.708910 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.708823    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:17.744901 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.744864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:17.900718 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.900655    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cf5wf" podStartSLOduration=2.781317096 podStartE2EDuration="3.900628035s" totalImagesPullingTime="1.04114492s" totalInitContainerRuntime="78.166019ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:17.696473344 +0000 UTC m=+81.030593265" watchObservedRunningTime="2026-07-31 14:49:17.900628035 +0000 UTC m=+81.234747955"
Jul 31 14:49:17.901391 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.901339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"]
Jul 31 14:49:17.903153 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:17.903130    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410bfc4b_b27a_4740_b7a9_ecfd14529b0b.slice/crio-b990c3ec3a11dcac7fbcd2db7cf3987db8e718ed4dc240fc53feb78c0d8ee52e WatchSource:0}: Error finding container b990c3ec3a11dcac7fbcd2db7cf3987db8e718ed4dc240fc53feb78c0d8ee52e: Status 404 returned error can't find the container with id b990c3ec3a11dcac7fbcd2db7cf3987db8e718ed4dc240fc53feb78c0d8ee52e
Jul 31 14:49:17.950881 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:17.950854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:49:18.091168 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.091076    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:18.664742 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.664704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"b990c3ec3a11dcac7fbcd2db7cf3987db8e718ed4dc240fc53feb78c0d8ee52e"}
Jul 31 14:49:18.993464 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.993366    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7756dfdb96-l8lj8"]
Jul 31 14:49:18.993819 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.993792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:18.995759 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.995735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:18.998552 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.998533    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 14:49:18.999543 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:18.999515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 14:49:19.000820 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.000271    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cvqcj0hi9ajhl\""
Jul 31 14:49:19.000820 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.000464    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 14:49:19.001513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.001036    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kmjdq\""
Jul 31 14:49:19.001513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.001262    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 14:49:19.012101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.012060    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7756dfdb96-l8lj8"]
Jul 31 14:49:19.074085 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.074042    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:19.118991 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.118956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a18a5d-30cb-408b-b700-f3586e60c275-audit-log\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-tls\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-metrics-server-audit-profiles\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119129    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4bm8\" (UniqueName: \"kubernetes.io/projected/b6a18a5d-30cb-408b-b700-f3586e60c275-kube-api-access-z4bm8\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119334 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-client-certs\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.119334 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.119208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-client-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.159496 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.159453    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:19.219690 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4bm8\" (UniqueName: \"kubernetes.io/projected/b6a18a5d-30cb-408b-b700-f3586e60c275-kube-api-access-z4bm8\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-client-certs\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-client-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a18a5d-30cb-408b-b700-f3586e60c275-audit-log\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-tls\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.219983 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.219923    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-metrics-server-audit-profiles\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.220427 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.220365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a18a5d-30cb-408b-b700-f3586e60c275-audit-log\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.220568 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.220535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.220901 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.220882    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a18a5d-30cb-408b-b700-f3586e60c275-metrics-server-audit-profiles\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.223006 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.222980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-client-ca-bundle\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.223116 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.223082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-tls\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.223116 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.223092    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a18a5d-30cb-408b-b700-f3586e60c275-secret-metrics-server-client-certs\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.229409 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.229375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4bm8\" (UniqueName: \"kubernetes.io/projected/b6a18a5d-30cb-408b-b700-f3586e60c275-kube-api-access-z4bm8\") pod \"metrics-server-7756dfdb96-l8lj8\" (UID: \"b6a18a5d-30cb-408b-b700-f3586e60c275\") " pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.310343 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.310257    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:19.318468 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.318437    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:49:19.322083 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.322052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.326216 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.325657    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 14:49:19.326216 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.325772    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 14:49:19.326216 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.326054    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 14:49:19.326216 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.326097    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 14:49:19.326507 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.326292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 14:49:19.326507 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.326469    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 14:49:19.326873 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.326855    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 14:49:19.327118 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.327096    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4mkg9\""
Jul 31 14:49:19.352513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.352481    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:49:19.421940 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.421911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.422150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.421949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.422150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.421995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.422150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.422038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.422150 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.422096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2xp\" (UniqueName: \"kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.422317 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.422212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.472698 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.472665    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7756dfdb96-l8lj8"]
Jul 31 14:49:19.523569 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.523770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.523770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.523770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.523770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.523770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.523745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2xp\" (UniqueName: \"kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.524857 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.524781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.524996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.524956    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.525099 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.525078    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.526576 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.526551    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.526734 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.526708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.533586 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.533558    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2xp\" (UniqueName: \"kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp\") pod \"console-5c4b89788d-qqsdm\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") " pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.647771 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.647735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:19.778285 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.778249    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:19.922407 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:19.922323    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:19.947803 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:19.947755    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a18a5d_30cb_408b_b700_f3586e60c275.slice/crio-3eb6343b2e787f398c797ad25b15347481c4bd2eb2c42f1f4fd33f3bc323b67f WatchSource:0}: Error finding container 3eb6343b2e787f398c797ad25b15347481c4bd2eb2c42f1f4fd33f3bc323b67f: Status 404 returned error can't find the container with id 3eb6343b2e787f398c797ad25b15347481c4bd2eb2c42f1f4fd33f3bc323b67f
Jul 31 14:49:20.107233 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.107196    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:49:20.111877 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:20.111833    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2deb64eb_cc99_4aa5_aae7_c323bd6029fa.slice/crio-049b1951a60878bfc519c3ecf065544d7981b95a234bd5fa9375a8c584e75139 WatchSource:0}: Error finding container 049b1951a60878bfc519c3ecf065544d7981b95a234bd5fa9375a8c584e75139: Status 404 returned error can't find the container with id 049b1951a60878bfc519c3ecf065544d7981b95a234bd5fa9375a8c584e75139
Jul 31 14:49:20.571622 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.571443    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-679dfb877-kg964"
Jul 31 14:49:20.675889 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.675853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b89788d-qqsdm" event={"ID":"2deb64eb-cc99-4aa5-aae7-c323bd6029fa","Type":"ContainerStarted","Data":"049b1951a60878bfc519c3ecf065544d7981b95a234bd5fa9375a8c584e75139"}
Jul 31 14:49:20.677228 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.677190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8" event={"ID":"b6a18a5d-30cb-408b-b700-f3586e60c275","Type":"ContainerStarted","Data":"3eb6343b2e787f398c797ad25b15347481c4bd2eb2c42f1f4fd33f3bc323b67f"}
Jul 31 14:49:20.679650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.679592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"61b5162f447869407ae6b45dbe8ddadc224a2125f32b038d28026727bd811e3a"}
Jul 31 14:49:20.679796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.679693    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"d8bc6db5e5925a478304ae913b537a9266a1af382b2ea98f7531b7562a1701eb"}
Jul 31 14:49:20.679796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.679726    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"32740d2ec878ba1dd06f1f6c451e9e815aacc83bcb81d2d7a7a0b8aeb9638827"}
Jul 31 14:49:20.894426 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:20.894377    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:21.085153 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:21.085066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:21.274385 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:21.274232    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:21.347936 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:21.347894    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:22.690935 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.690828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8" event={"ID":"b6a18a5d-30cb-408b-b700-f3586e60c275","Type":"ContainerStarted","Data":"59daa8feacb547b9e117d86d0bcf46599103a85f5872b6f91710fd1f980041f6"}
Jul 31 14:49:22.694122 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.694084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"b8eba307f9d667524efbba4995715cd41793b0e1a1da670ca8d1418c19f61641"}
Jul 31 14:49:22.694264 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.694159    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"0b53af3fe1e9fee50c2417c2efaf85c1ec538eb2322bfddff236b1e1fe31de39"}
Jul 31 14:49:22.694264 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.694203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" event={"ID":"410bfc4b-b27a-4740-b7a9-ecfd14529b0b","Type":"ContainerStarted","Data":"9c2f087f3e6cddf42ba5d2f40df709d399acbade3b3c5b0d623240f01356f5de"}
Jul 31 14:49:22.694264 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.694238    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:22.715074 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.715004    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8" podStartSLOduration=2.959330629 podStartE2EDuration="4.714979485s" totalImagesPullingTime="1.755648856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:22.712654145 +0000 UTC m=+86.046774060" watchObservedRunningTime="2026-07-31 14:49:22.714979485 +0000 UTC m=+86.049099407"
Jul 31 14:49:22.739296 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.739233    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq" podStartSLOduration=2.158121054 podStartE2EDuration="5.739213251s" totalImagesPullingTime="3.581092197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:22.737541588 +0000 UTC m=+86.071661513" watchObservedRunningTime="2026-07-31 14:49:22.739213251 +0000 UTC m=+86.073333176"
Jul 31 14:49:22.965457 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:22.964621    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" podUID="87744021-57ed-452d-8187-173d2f849f42" containerName="registry" containerID="cri-o://680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61" gracePeriod=30
Jul 31 14:49:23.484059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.484033    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:49:23.566426 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566382    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566459    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566494    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cch6f\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566518    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566538    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566559    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566577    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.566650 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566630    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration\") pod \"87744021-57ed-452d-8187-173d2f849f42\" (UID: \"87744021-57ed-452d-8187-173d2f849f42\") "
Jul 31 14:49:23.567016 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.566930    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:49:23.567155 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.567112    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:49:23.569401 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.569368    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:49:23.569545 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.569452    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:49:23.569545 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.569522    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f" (OuterVolumeSpecName: "kube-api-access-cch6f") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "kube-api-access-cch6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:49:23.569693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.569616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:49:23.569693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.569649    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:49:23.576493 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.576452    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "87744021-57ed-452d-8187-173d2f849f42" (UID: "87744021-57ed-452d-8187-173d2f849f42"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:49:23.667869 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667823    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-trusted-ca\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.667869 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667858    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-registry-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.667869 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667869    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87744021-57ed-452d-8187-173d2f849f42-ca-trust-extracted\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.667869 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667880    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-image-registry-private-configuration\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.668149 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667891    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87744021-57ed-452d-8187-173d2f849f42-registry-certificates\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.668149 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667899    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87744021-57ed-452d-8187-173d2f849f42-installation-pull-secrets\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.668149 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667907    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cch6f\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-kube-api-access-cch6f\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.668149 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.667915    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87744021-57ed-452d-8187-173d2f849f42-bound-sa-token\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:49:23.700202 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.700164    2575 generic.go:356] "Generic (PLEG): container finished" podID="87744021-57ed-452d-8187-173d2f849f42" containerID="680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61" exitCode=0
Jul 31 14:49:23.700719 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.700221    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5"
Jul 31 14:49:23.700719 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.700252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" event={"ID":"87744021-57ed-452d-8187-173d2f849f42","Type":"ContainerDied","Data":"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"}
Jul 31 14:49:23.700719 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.700307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55fdf7c5b-2wgc5" event={"ID":"87744021-57ed-452d-8187-173d2f849f42","Type":"ContainerDied","Data":"e62bfe34c4b83fa7236777f0812030ed7787e56eb4684dbf34db6deca75059e2"}
Jul 31 14:49:23.700719 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.700336    2575 scope.go:117] "RemoveContainer" containerID="680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"
Jul 31 14:49:23.702047 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.702018    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b89788d-qqsdm" event={"ID":"2deb64eb-cc99-4aa5-aae7-c323bd6029fa","Type":"ContainerStarted","Data":"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"}
Jul 31 14:49:23.713302 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.713277    2575 scope.go:117] "RemoveContainer" containerID="680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"
Jul 31 14:49:23.713696 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:49:23.713674    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61\": container with ID starting with 680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61 not found: ID does not exist" containerID="680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"
Jul 31 14:49:23.713747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.713706    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61"} err="failed to get container status \"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61\": rpc error: code = NotFound desc = could not find container \"680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61\": container with ID starting with 680452909a2e7f5c1379819ee9aea5a21bd357c26b13ebb6ce848b95ea7deb61 not found: ID does not exist"
Jul 31 14:49:23.730344 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.730279    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c4b89788d-qqsdm" podStartSLOduration=1.441405659 podStartE2EDuration="4.730259535s" totalImagesPullingTime="3.288853876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:23.728984199 +0000 UTC m=+87.063104144" watchObservedRunningTime="2026-07-31 14:49:23.730259535 +0000 UTC m=+87.064379457"
Jul 31 14:49:23.750533 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.750496    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:49:23.756519 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:23.756492    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55fdf7c5b-2wgc5"]
Jul 31 14:49:25.201316 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:25.201278    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87744021-57ed-452d-8187-173d2f849f42" path="/var/lib/kubelet/pods/87744021-57ed-452d-8187-173d2f849f42/volumes"
Jul 31 14:49:28.709085 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:28.709054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7fb9c6997f-vknzq"
Jul 31 14:49:29.093408 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.093325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:29.190838 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.190800    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:29.648464 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.648415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:29.648698 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.648500    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:29.654432 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.654402    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:29.698532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.698492    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:49:29.698812 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.698799    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87744021-57ed-452d-8187-173d2f849f42" containerName="registry"
Jul 31 14:49:29.698865 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.698813    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87744021-57ed-452d-8187-173d2f849f42" containerName="registry"
Jul 31 14:49:29.698865 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.698863    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87744021-57ed-452d-8187-173d2f849f42" containerName="registry"
Jul 31 14:49:29.703156 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.703122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.712903 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.712877    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 31 14:49:29.715671 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.715641    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:49:29.732821 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.732789    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:49:29.816432 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgbxt\" (UniqueName: \"kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816638 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816696 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816696 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816786 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816901 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.816940 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.816912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918390 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918390 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918628 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgbxt\" (UniqueName: \"kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918628 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918628 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918628 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.918628 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.918522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.919287 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.919255    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.919400 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.919371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.919576 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.919544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.919576 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.919544    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.922003 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.921980    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.922194 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.922168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:29.928883 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:29.928854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgbxt\" (UniqueName: \"kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt\") pod \"console-6975686468-5ssgm\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") " pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:30.013053 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:30.013011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:30.166907 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:30.166859    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:49:30.170844 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:49:30.170805    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779f0752_760a_4049_95e5_5b9ca45823bf.slice/crio-e6bb82d7c7d2a566e10c2919d64445900515423c35612d077f516366e82b808f WatchSource:0}: Error finding container e6bb82d7c7d2a566e10c2919d64445900515423c35612d077f516366e82b808f: Status 404 returned error can't find the container with id e6bb82d7c7d2a566e10c2919d64445900515423c35612d077f516366e82b808f
Jul 31 14:49:30.733361 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:30.733319    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975686468-5ssgm" event={"ID":"779f0752-760a-4049-95e5-5b9ca45823bf","Type":"ContainerStarted","Data":"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"}
Jul 31 14:49:30.733882 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:30.733382    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975686468-5ssgm" event={"ID":"779f0752-760a-4049-95e5-5b9ca45823bf","Type":"ContainerStarted","Data":"e6bb82d7c7d2a566e10c2919d64445900515423c35612d077f516366e82b808f"}
Jul 31 14:49:30.757550 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:30.757496    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6975686468-5ssgm" podStartSLOduration=1.757478951 podStartE2EDuration="1.757478951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:30.756277765 +0000 UTC m=+94.090397699" watchObservedRunningTime="2026-07-31 14:49:30.757478951 +0000 UTC m=+94.091598874"
Jul 31 14:49:32.198680 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:32.198633    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:32.277428 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:32.277389    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:33.519487 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:33.519449    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:34.592242 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:34.592211    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dt4ln"
Jul 31 14:49:35.839227 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:35.839194    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:35.945843 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:35.945807    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:39.212109 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.212079    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zx94w_22d806cf-8c2e-4a36-a22f-f7e32bb1f619/serve-healthcheck-canary/0.log"
Jul 31 14:49:39.309910 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.309877    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:39.310712 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.310680    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:39.310787 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.310745    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:39.421021 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.420978    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:39.553662 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.553552    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:39.660350 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:39.660315    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:40.013864 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:40.013828    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:40.014084 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:40.013898    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:40.019093 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:40.019066    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:40.777374 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:40.777345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:49:40.903056 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:40.903024    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:49:41.969046 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:41.969011    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:42.411007 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:42.410976    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:43.126994 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:43.126953    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:43.228719 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:43.228683    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:44.112801 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:44.112763    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:44.212909 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:44.212872    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:47.051900 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:47.051865    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:47.154243 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:47.154205    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:47.242289 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:47.242259    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:50.572883 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:50.572846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:50.677098 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:50.677066    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:50.872371 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:50.872334    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:50.934939 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:50.934901    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:49:59.316198 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:59.316161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:49:59.320330 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:49:59.320302    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7756dfdb96-l8lj8"
Jul 31 14:50:05.922443 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:05.922395    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c4b89788d-qqsdm" podUID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" containerName="console" containerID="cri-o://bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458" gracePeriod=15
Jul 31 14:50:06.192872 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.192840    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:50:06.223169 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223169 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223167    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223397 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223193    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223397 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223228    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223397 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223261    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223397 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223306    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk2xp\" (UniqueName: \"kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp\") pod \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\" (UID: \"2deb64eb-cc99-4aa5-aae7-c323bd6029fa\") "
Jul 31 14:50:06.223889 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223654    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:06.223889 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:06.223889 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.223675    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config" (OuterVolumeSpecName: "console-config") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:06.225744 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.225712    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:06.225837 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.225777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:06.225837 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.225769    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp" (OuterVolumeSpecName: "kube-api-access-mk2xp") pod "2deb64eb-cc99-4aa5-aae7-c323bd6029fa" (UID: "2deb64eb-cc99-4aa5-aae7-c323bd6029fa"). InnerVolumeSpecName "kube-api-access-mk2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:06.324753 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324711    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2xp\" (UniqueName: \"kubernetes.io/projected/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-kube-api-access-mk2xp\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.324753 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324745    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-service-ca\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.324753 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324755    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.324753 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324764    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-oauth-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.325016 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324773    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.325016 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.324782    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2deb64eb-cc99-4aa5-aae7-c323bd6029fa-console-oauth-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:50:06.883461 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.883420    2575 generic.go:356] "Generic (PLEG): container finished" podID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" containerID="bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458" exitCode=0
Jul 31 14:50:06.883688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.883490    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c4b89788d-qqsdm"
Jul 31 14:50:06.883688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.883511    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b89788d-qqsdm" event={"ID":"2deb64eb-cc99-4aa5-aae7-c323bd6029fa","Type":"ContainerDied","Data":"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"}
Jul 31 14:50:06.883688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.883588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c4b89788d-qqsdm" event={"ID":"2deb64eb-cc99-4aa5-aae7-c323bd6029fa","Type":"ContainerDied","Data":"049b1951a60878bfc519c3ecf065544d7981b95a234bd5fa9375a8c584e75139"}
Jul 31 14:50:06.883688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.883630    2575 scope.go:117] "RemoveContainer" containerID="bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"
Jul 31 14:50:06.893008 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.892990    2575 scope.go:117] "RemoveContainer" containerID="bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"
Jul 31 14:50:06.893354 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:50:06.893333    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458\": container with ID starting with bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458 not found: ID does not exist" containerID="bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"
Jul 31 14:50:06.893405 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.893366    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458"} err="failed to get container status \"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458\": rpc error: code = NotFound desc = could not find container \"bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458\": container with ID starting with bf51ad537ee3249217e9fec44766761f85835ebf47ce090d49f453b8934b5458 not found: ID does not exist"
Jul 31 14:50:06.905900 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.905866    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:50:06.911111 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:06.911083    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c4b89788d-qqsdm"]
Jul 31 14:50:07.201526 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:07.201444    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" path="/var/lib/kubelet/pods/2deb64eb-cc99-4aa5-aae7-c323bd6029fa/volumes"
Jul 31 14:50:13.070951 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:13.070912    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:33.827439 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:33.827397    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:34.017553 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:34.017518    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:35.356739 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:35.356699    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:41.788426 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.788386    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:50:41.788944 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.788827    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" containerName="console"
Jul 31 14:50:41.788944 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.788844    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" containerName="console"
Jul 31 14:50:41.788944 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.788909    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2deb64eb-cc99-4aa5-aae7-c323bd6029fa" containerName="console"
Jul 31 14:50:41.791646 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.791624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.803864 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.803832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:50:41.854956 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.854917    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:41.913846 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.913987 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.913987 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.913987 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.913987 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.913987 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.913954    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.914167 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.914002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqq86\" (UniqueName: \"kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:41.976290 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:41.976242    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:42.015013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.014976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015013 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015309 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015043    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015309 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015309 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015309 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqq86\" (UniqueName: \"kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015309 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.015888 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.015861    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.016057 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.016030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.016175 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.016161    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.016238 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.016181    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.018907 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.018869    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.019057 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.018973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.025404 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.025369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqq86\" (UniqueName: \"kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86\") pod \"console-6b8d695649-phwsm\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") " pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.100711 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.100567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:42.288086 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:42.287993    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:50:42.292813 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:50:42.292769    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8ce706_6e72_4984_8adb_be47771823f5.slice/crio-973a3205b280a49daf107b5d9d845b0851594c2ea81be1f6c8da8e6e7aa3ea32 WatchSource:0}: Error finding container 973a3205b280a49daf107b5d9d845b0851594c2ea81be1f6c8da8e6e7aa3ea32: Status 404 returned error can't find the container with id 973a3205b280a49daf107b5d9d845b0851594c2ea81be1f6c8da8e6e7aa3ea32
Jul 31 14:50:43.025200 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:43.025144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d695649-phwsm" event={"ID":"9a8ce706-6e72-4984-8adb-be47771823f5","Type":"ContainerStarted","Data":"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"}
Jul 31 14:50:43.025619 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:43.025250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d695649-phwsm" event={"ID":"9a8ce706-6e72-4984-8adb-be47771823f5","Type":"ContainerStarted","Data":"973a3205b280a49daf107b5d9d845b0851594c2ea81be1f6c8da8e6e7aa3ea32"}
Jul 31 14:50:43.055203 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:43.055143    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b8d695649-phwsm" podStartSLOduration=2.055125483 podStartE2EDuration="2.055125483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:43.053416683 +0000 UTC m=+166.387536605" watchObservedRunningTime="2026-07-31 14:50:43.055125483 +0000 UTC m=+166.389245404"
Jul 31 14:50:45.430477 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:45.430434    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:52.101434 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:52.101337    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:52.101434 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:52.101394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:52.106444 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:52.106415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:53.066104 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:53.066069    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:50:53.121148 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:53.121112    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:50:53.258863 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:53.258822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:53.359859 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:53.359825    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:50:53.421452 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:50:53.421424    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:09.423315 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:09.423280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:09.493075 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:09.493047    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:18.141520 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.141476    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6975686468-5ssgm" podUID="779f0752-760a-4049-95e5-5b9ca45823bf" containerName="console" containerID="cri-o://b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505" gracePeriod=15
Jul 31 14:51:18.382903 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.382866    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:51:18.416680 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416553    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgbxt\" (UniqueName: \"kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416680 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416616    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416680 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416643    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416705    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416767    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416791    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.416942 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.416818    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config\") pod \"779f0752-760a-4049-95e5-5b9ca45823bf\" (UID: \"779f0752-760a-4049-95e5-5b9ca45823bf\") "
Jul 31 14:51:18.417132 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.417053    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:18.417132 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.417100    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:18.417281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.417155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:18.417357 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.417327    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config" (OuterVolumeSpecName: "console-config") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:18.419177 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.419147    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt" (OuterVolumeSpecName: "kube-api-access-wgbxt") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "kube-api-access-wgbxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:18.419177 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.419157    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:18.419368 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.419351    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "779f0752-760a-4049-95e5-5b9ca45823bf" (UID: "779f0752-760a-4049-95e5-5b9ca45823bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:18.518344 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518309    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-service-ca\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518344 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518339    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-oauth-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518344 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518349    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-console-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518344 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518358    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgbxt\" (UniqueName: \"kubernetes.io/projected/779f0752-760a-4049-95e5-5b9ca45823bf-kube-api-access-wgbxt\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518649 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518366    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-oauth-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518649 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518374    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/779f0752-760a-4049-95e5-5b9ca45823bf-trusted-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:18.518649 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:18.518383    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/779f0752-760a-4049-95e5-5b9ca45823bf-console-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:51:19.159352 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.159313    2575 generic.go:356] "Generic (PLEG): container finished" podID="779f0752-760a-4049-95e5-5b9ca45823bf" containerID="b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505" exitCode=0
Jul 31 14:51:19.159821 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.159369    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6975686468-5ssgm"
Jul 31 14:51:19.159821 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.159397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975686468-5ssgm" event={"ID":"779f0752-760a-4049-95e5-5b9ca45823bf","Type":"ContainerDied","Data":"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"}
Jul 31 14:51:19.159821 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.159536    2575 scope.go:117] "RemoveContainer" containerID="b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"
Jul 31 14:51:19.159969 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.159868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6975686468-5ssgm" event={"ID":"779f0752-760a-4049-95e5-5b9ca45823bf","Type":"ContainerDied","Data":"e6bb82d7c7d2a566e10c2919d64445900515423c35612d077f516366e82b808f"}
Jul 31 14:51:19.173261 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.173236    2575 scope.go:117] "RemoveContainer" containerID="b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"
Jul 31 14:51:19.173748 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:51:19.173713    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505\": container with ID starting with b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505 not found: ID does not exist" containerID="b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"
Jul 31 14:51:19.173813 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.173758    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505"} err="failed to get container status \"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505\": rpc error: code = NotFound desc = could not find container \"b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505\": container with ID starting with b2a42420bdf699daabe7c82ab7b0dbaa70c383c597e02ba708a88e930e8d2505 not found: ID does not exist"
Jul 31 14:51:19.186613 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.186564    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:51:19.191052 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.191021    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6975686468-5ssgm"]
Jul 31 14:51:19.201697 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.201662    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779f0752-760a-4049-95e5-5b9ca45823bf" path="/var/lib/kubelet/pods/779f0752-760a-4049-95e5-5b9ca45823bf/volumes"
Jul 31 14:51:19.731350 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:19.731318    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:49.111894 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:49.111859    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:49.212926 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:49.212885    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:53.785087 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.785046    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:51:53.785568 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.785432    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779f0752-760a-4049-95e5-5b9ca45823bf" containerName="console"
Jul 31 14:51:53.785568 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.785450    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="779f0752-760a-4049-95e5-5b9ca45823bf" containerName="console"
Jul 31 14:51:53.785568 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.785542    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="779f0752-760a-4049-95e5-5b9ca45823bf" containerName="console"
Jul 31 14:51:53.788276 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.788249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.801283 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.801255    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:51:53.853162 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.853124    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:53.904286 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904237    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904286 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zh9p\" (UniqueName: \"kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904473 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904473 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904473 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904473 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.904627 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.904485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:53.916372 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:53.916341    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:51:54.005679 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005642    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.005791 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.005791 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.005791 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.005926 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.005926 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.006020 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.005957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zh9p\" (UniqueName: \"kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.006525 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.006493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.006689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.006657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.006689 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.006683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.006823 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.006798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.008317 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.008289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.008408 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.008295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.016389 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.016360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zh9p\" (UniqueName: \"kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p\") pod \"console-64d5d458b8-hjwz9\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") " pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.098036 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.097933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:51:54.237420 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.237383    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:51:54.240319 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:51:54.240283    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde72eadc_12b7_4955_aac6_cdac6e594678.slice/crio-bf9ae5f75d65a556bd48e4b123ec9dc43990d47d72225e2c59bf98e5285416e8 WatchSource:0}: Error finding container bf9ae5f75d65a556bd48e4b123ec9dc43990d47d72225e2c59bf98e5285416e8: Status 404 returned error can't find the container with id bf9ae5f75d65a556bd48e4b123ec9dc43990d47d72225e2c59bf98e5285416e8
Jul 31 14:51:54.290974 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:54.290943    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5d458b8-hjwz9" event={"ID":"de72eadc-12b7-4955-aac6-cdac6e594678","Type":"ContainerStarted","Data":"bf9ae5f75d65a556bd48e4b123ec9dc43990d47d72225e2c59bf98e5285416e8"}
Jul 31 14:51:55.296011 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:55.295975    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5d458b8-hjwz9" event={"ID":"de72eadc-12b7-4955-aac6-cdac6e594678","Type":"ContainerStarted","Data":"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"}
Jul 31 14:51:55.315101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:51:55.315040    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64d5d458b8-hjwz9" podStartSLOduration=2.315023177 podStartE2EDuration="2.315023177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:55.313704268 +0000 UTC m=+238.647824191" watchObservedRunningTime="2026-07-31 14:51:55.315023177 +0000 UTC m=+238.649143098"
Jul 31 14:52:01.113396 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:01.113364    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:01.213693 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:01.213653    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:04.098330 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.098273    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:52:04.098781 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.098375    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:52:04.103285 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.103259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:52:04.340387 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.340350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:52:04.387929 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.387892    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:52:04.506678 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.506594    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:04.608110 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.608079    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:04.661465 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:04.661390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:05.835432 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:05.835392    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:05.934932 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:05.934897    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:52:29.413790 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.413699    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b8d695649-phwsm" podUID="9a8ce706-6e72-4984-8adb-be47771823f5" containerName="console" containerID="cri-o://12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb" gracePeriod=15
Jul 31 14:52:29.660563 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.660533    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:52:29.776905 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.776811    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.776905 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.776864    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.776905 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.776891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.777170 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.776936    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.777170 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.776969    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqq86\" (UniqueName: \"kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.777170 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777139    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.777320 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777212    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca\") pod \"9a8ce706-6e72-4984-8adb-be47771823f5\" (UID: \"9a8ce706-6e72-4984-8adb-be47771823f5\") "
Jul 31 14:52:29.777320 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:52:29.777320 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777293    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config" (OuterVolumeSpecName: "console-config") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:52:29.777461 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777448    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-trusted-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.777511 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777465    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-console-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.777623 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777574    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:52:29.777759 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.777655    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:52:29.779373 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.779339    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86" (OuterVolumeSpecName: "kube-api-access-hqq86") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "kube-api-access-hqq86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:52:29.779373 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.779336    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:52:29.779546 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.779383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a8ce706-6e72-4984-8adb-be47771823f5" (UID: "9a8ce706-6e72-4984-8adb-be47771823f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:52:29.878752 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.878711    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.878752 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.878742    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqq86\" (UniqueName: \"kubernetes.io/projected/9a8ce706-6e72-4984-8adb-be47771823f5-kube-api-access-hqq86\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.878752 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.878752    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-oauth-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.878752 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.878761    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8ce706-6e72-4984-8adb-be47771823f5-service-ca\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:29.878752 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:29.878770    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8ce706-6e72-4984-8adb-be47771823f5-console-oauth-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:52:30.430620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.430560    2575 generic.go:356] "Generic (PLEG): container finished" podID="9a8ce706-6e72-4984-8adb-be47771823f5" containerID="12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb" exitCode=0
Jul 31 14:52:30.431163 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.430628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b8d695649-phwsm"
Jul 31 14:52:30.431163 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.430656    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d695649-phwsm" event={"ID":"9a8ce706-6e72-4984-8adb-be47771823f5","Type":"ContainerDied","Data":"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"}
Jul 31 14:52:30.431163 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.430708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b8d695649-phwsm" event={"ID":"9a8ce706-6e72-4984-8adb-be47771823f5","Type":"ContainerDied","Data":"973a3205b280a49daf107b5d9d845b0851594c2ea81be1f6c8da8e6e7aa3ea32"}
Jul 31 14:52:30.431163 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.430728    2575 scope.go:117] "RemoveContainer" containerID="12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"
Jul 31 14:52:30.439426 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.439403    2575 scope.go:117] "RemoveContainer" containerID="12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"
Jul 31 14:52:30.439802 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:52:30.439782    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb\": container with ID starting with 12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb not found: ID does not exist" containerID="12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"
Jul 31 14:52:30.439861 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.439814    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb"} err="failed to get container status \"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb\": rpc error: code = NotFound desc = could not find container \"12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb\": container with ID starting with 12971a74b35f9a13c98b12830076e4a17aabbb17c19a044059a05eea253dcadb not found: ID does not exist"
Jul 31 14:52:30.455829 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.455776    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:52:30.456029 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:30.455845    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b8d695649-phwsm"]
Jul 31 14:52:31.201404 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:31.201370    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8ce706-6e72-4984-8adb-be47771823f5" path="/var/lib/kubelet/pods/9a8ce706-6e72-4984-8adb-be47771823f5/volumes"
Jul 31 14:52:57.059837 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:52:57.059808    2575 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:53:47.496436 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:53:47.496400    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:14.243518 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:14.243442    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:14.347932 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:14.347899    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:14.442620 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:14.442551    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:15.494950 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.494913    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"]
Jul 31 14:54:15.495338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.495218    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8ce706-6e72-4984-8adb-be47771823f5" containerName="console"
Jul 31 14:54:15.495338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.495230    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8ce706-6e72-4984-8adb-be47771823f5" containerName="console"
Jul 31 14:54:15.495338 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.495277    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8ce706-6e72-4984-8adb-be47771823f5" containerName="console"
Jul 31 14:54:15.498437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.498417    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.502259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.502225    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 31 14:54:15.502259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.502255    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 31 14:54:15.503297 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.503279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4fkwq\""
Jul 31 14:54:15.524944 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.524908    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"]
Jul 31 14:54:15.603592 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.603554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsjx7\" (UniqueName: \"kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.603796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.603689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.603796 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.603732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.704667 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.704624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.704839 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.704680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsjx7\" (UniqueName: \"kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.704839 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.704724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.705145 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.705117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.705183 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.705154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.714855 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.714820    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsjx7\" (UniqueName: \"kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.808140 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.808035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:15.960856 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.960818    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"]
Jul 31 14:54:15.964046 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:54:15.964015    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab40ed73_18f7_4e5e_a741_6bfee5a699cf.slice/crio-e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223 WatchSource:0}: Error finding container e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223: Status 404 returned error can't find the container with id e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223
Jul 31 14:54:15.965883 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:15.965863    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:54:16.831394 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:16.831336    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph" event={"ID":"ab40ed73-18f7-4e5e-a741-6bfee5a699cf","Type":"ContainerStarted","Data":"e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223"}
Jul 31 14:54:21.855007 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:21.854959    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerID="f7de2e751de96a0109e6a2dceede306c66ccf2de5f91427de63cf653dff347e4" exitCode=0
Jul 31 14:54:21.855402 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:21.855042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph" event={"ID":"ab40ed73-18f7-4e5e-a741-6bfee5a699cf","Type":"ContainerDied","Data":"f7de2e751de96a0109e6a2dceede306c66ccf2de5f91427de63cf653dff347e4"}
Jul 31 14:54:24.868489 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:24.868441    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerID="758df4104395328faf1a9e2d48d9a83a88e8e83d54775117f7e2c117469f2cfb" exitCode=0
Jul 31 14:54:24.868900 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:24.868521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph" event={"ID":"ab40ed73-18f7-4e5e-a741-6bfee5a699cf","Type":"ContainerDied","Data":"758df4104395328faf1a9e2d48d9a83a88e8e83d54775117f7e2c117469f2cfb"}
Jul 31 14:54:27.077148 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.077099    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f898d6645-h46kp"]
Jul 31 14:54:27.100186 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.100156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.103522 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.103481    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f898d6645-h46kp"]
Jul 31 14:54:27.125228 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.125188    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:27.208651 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208578    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-oauth-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.208651 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-service-ca\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.208884 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208684    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.208884 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.208884 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-oauth-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.208884 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mctmh\" (UniqueName: \"kubernetes.io/projected/6a7f172c-249e-43d8-b95a-9bda31c36d40-kube-api-access-mctmh\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.209067 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.208951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-trusted-ca-bundle\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.227273 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.227233    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:27.310396 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-trusted-ca-bundle\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310396 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-oauth-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310708 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-service-ca\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310708 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310708 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310505    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310708 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-oauth-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.310708 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.310572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mctmh\" (UniqueName: \"kubernetes.io/projected/6a7f172c-249e-43d8-b95a-9bda31c36d40-kube-api-access-mctmh\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.311424 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.311372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-trusted-ca-bundle\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.311748 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.311484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.311913 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.311890    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-service-ca\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.311986 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.311929    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a7f172c-249e-43d8-b95a-9bda31c36d40-oauth-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.313747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.313722    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-serving-cert\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.313863 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.313802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a7f172c-249e-43d8-b95a-9bda31c36d40-console-oauth-config\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.320736 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.320706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mctmh\" (UniqueName: \"kubernetes.io/projected/6a7f172c-249e-43d8-b95a-9bda31c36d40-kube-api-access-mctmh\") pod \"console-6f898d6645-h46kp\" (UID: \"6a7f172c-249e-43d8-b95a-9bda31c36d40\") " pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.412940 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.412898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:27.592678 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.592651    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f898d6645-h46kp"]
Jul 31 14:54:27.595367 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:54:27.595329    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7f172c_249e_43d8_b95a_9bda31c36d40.slice/crio-60c41ba622487dd55dde5759dbce9dbe0cf22f09dcec0b6743aa81df53b38f89 WatchSource:0}: Error finding container 60c41ba622487dd55dde5759dbce9dbe0cf22f09dcec0b6743aa81df53b38f89: Status 404 returned error can't find the container with id 60c41ba622487dd55dde5759dbce9dbe0cf22f09dcec0b6743aa81df53b38f89
Jul 31 14:54:27.882677 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.882620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f898d6645-h46kp" event={"ID":"6a7f172c-249e-43d8-b95a-9bda31c36d40","Type":"ContainerStarted","Data":"c06284bf8906bbf69459939fee106342984e3b4ed7f14e5001dfbcf035f60c22"}
Jul 31 14:54:27.882887 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.882760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f898d6645-h46kp" event={"ID":"6a7f172c-249e-43d8-b95a-9bda31c36d40","Type":"ContainerStarted","Data":"60c41ba622487dd55dde5759dbce9dbe0cf22f09dcec0b6743aa81df53b38f89"}
Jul 31 14:54:27.902360 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:27.902301    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f898d6645-h46kp" podStartSLOduration=0.902284122 podStartE2EDuration="902.284122ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:27.901110744 +0000 UTC m=+391.235230667" watchObservedRunningTime="2026-07-31 14:54:27.902284122 +0000 UTC m=+391.236404044"
Jul 31 14:54:31.901253 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:31.901215    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerID="c3a13461c53fda7b4b441ad7388b755f52ba3bd53fed32596041cc3b5305dde1" exitCode=0
Jul 31 14:54:31.901695 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:31.901308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph" event={"ID":"ab40ed73-18f7-4e5e-a741-6bfee5a699cf","Type":"ContainerDied","Data":"c3a13461c53fda7b4b441ad7388b755f52ba3bd53fed32596041cc3b5305dde1"}
Jul 31 14:54:33.032986 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.032952    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:33.063278 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.063239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsjx7\" (UniqueName: \"kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7\") pod \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") "
Jul 31 14:54:33.063450 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.063290    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util\") pod \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") "
Jul 31 14:54:33.063450 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.063342    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle\") pod \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\" (UID: \"ab40ed73-18f7-4e5e-a741-6bfee5a699cf\") "
Jul 31 14:54:33.064106 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.064074    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle" (OuterVolumeSpecName: "bundle") pod "ab40ed73-18f7-4e5e-a741-6bfee5a699cf" (UID: "ab40ed73-18f7-4e5e-a741-6bfee5a699cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:54:33.065777 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.065744    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7" (OuterVolumeSpecName: "kube-api-access-rsjx7") pod "ab40ed73-18f7-4e5e-a741-6bfee5a699cf" (UID: "ab40ed73-18f7-4e5e-a741-6bfee5a699cf"). InnerVolumeSpecName "kube-api-access-rsjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:54:33.069203 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.069163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util" (OuterVolumeSpecName: "util") pod "ab40ed73-18f7-4e5e-a741-6bfee5a699cf" (UID: "ab40ed73-18f7-4e5e-a741-6bfee5a699cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:54:33.164241 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.164148    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsjx7\" (UniqueName: \"kubernetes.io/projected/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-kube-api-access-rsjx7\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:54:33.164241 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.164183    2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-util\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:54:33.164241 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.164192    2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab40ed73-18f7-4e5e-a741-6bfee5a699cf-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:54:33.911173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.911130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph" event={"ID":"ab40ed73-18f7-4e5e-a741-6bfee5a699cf","Type":"ContainerDied","Data":"e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223"}
Jul 31 14:54:33.911173 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.911178    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b52cxph"
Jul 31 14:54:33.911386 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:33.911194    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83526c693ccbbf028dd8e50fb1b67e374a2bf3aff9e677b07b9035230d25223"
Jul 31 14:54:37.153750 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.153712    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:37.274996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.274961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:37.413363 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.413261    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:37.413363 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.413332    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:37.418271 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.418241    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:37.931143 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:37.931116    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f898d6645-h46kp"
Jul 31 14:54:38.042914 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:38.042881    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:54:38.340511 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:38.340406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:38.552442 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:38.552403    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:39.327700 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.327657    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"]
Jul 31 14:54:39.328043 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328026    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="util"
Jul 31 14:54:39.328123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328045    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="util"
Jul 31 14:54:39.328123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328059    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="pull"
Jul 31 14:54:39.328123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328067    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="pull"
Jul 31 14:54:39.328123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328082    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="extract"
Jul 31 14:54:39.328123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328092    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="extract"
Jul 31 14:54:39.328345 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.328175    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab40ed73-18f7-4e5e-a741-6bfee5a699cf" containerName="extract"
Jul 31 14:54:39.330210 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.330186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.335200 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.335180    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 31 14:54:39.335497 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.335478    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zgzzv\""
Jul 31 14:54:39.335497 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.335489    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 31 14:54:39.335663 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.335495    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 31 14:54:39.353416 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.353389    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"]
Jul 31 14:54:39.408158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.408121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.408158 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.408157    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vg2\" (UniqueName: \"kubernetes.io/projected/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-kube-api-access-j6vg2\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.509359 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.509323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.509359 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.509362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vg2\" (UniqueName: \"kubernetes.io/projected/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-kube-api-access-j6vg2\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.511977 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.511943    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.519657 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.519624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vg2\" (UniqueName: \"kubernetes.io/projected/4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7-kube-api-access-j6vg2\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f\" (UID: \"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.640549 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.640505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:39.789292 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.789265    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"]
Jul 31 14:54:39.792128 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:54:39.792093    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4204fd9a_b2ed_4a73_8bfd_2b9035a60aa7.slice/crio-4d9e524c6577159f576fae9422975dcf569715ad3bcb5633f3b57ee436b2cb4d WatchSource:0}: Error finding container 4d9e524c6577159f576fae9422975dcf569715ad3bcb5633f3b57ee436b2cb4d: Status 404 returned error can't find the container with id 4d9e524c6577159f576fae9422975dcf569715ad3bcb5633f3b57ee436b2cb4d
Jul 31 14:54:39.937063 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:39.936967    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f" event={"ID":"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7","Type":"ContainerStarted","Data":"4d9e524c6577159f576fae9422975dcf569715ad3bcb5633f3b57ee436b2cb4d"}
Jul 31 14:54:42.911122 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:42.911083    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:42.955765 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:42.955729    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f" event={"ID":"4204fd9a-b2ed-4a73-8bfd-2b9035a60aa7","Type":"ContainerStarted","Data":"8a6a94fc07457d6c3161b87e8447152cde5e4fef138cf8e0e1e5e988cd3e0003"}
Jul 31 14:54:42.955970 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:42.955815    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:54:42.980377 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:42.980315    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f" podStartSLOduration=1.388594043 podStartE2EDuration="3.980298045s" totalImagesPullingTime="2.591704002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:54:42.977778381 +0000 UTC m=+406.311898304" watchObservedRunningTime="2026-07-31 14:54:42.980298045 +0000 UTC m=+406.314417967"
Jul 31 14:54:43.215647 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.215528    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.374617 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.374559    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.484090 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.484004    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.572286 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.572254    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.710653 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.710615    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.814381 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.814272    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:43.917165 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:43.917135    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:44.121123 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:44.121088    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:44.222001 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:44.221943    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:44.480142 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:44.480043    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:54:44.594391 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:54:44.594352    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:55:03.062475 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.062423    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64d5d458b8-hjwz9" podUID="de72eadc-12b7-4955-aac6-cdac6e594678" containerName="console" containerID="cri-o://cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d" gracePeriod=15
Jul 31 14:55:03.338993 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.338911    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:55:03.389023 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.388988    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389023 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389031    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389094    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zh9p\" (UniqueName: \"kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389194    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389280 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389275    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389467 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389322    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle\") pod \"de72eadc-12b7-4955-aac6-cdac6e594678\" (UID: \"de72eadc-12b7-4955-aac6-cdac6e594678\") "
Jul 31 14:55:03.389555 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389518    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca" (OuterVolumeSpecName: "service-ca") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:55:03.389555 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389516    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:55:03.389672 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389580    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config" (OuterVolumeSpecName: "console-config") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:55:03.389881 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.389843    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:55:03.391532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.391492    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:55:03.391532 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.391506    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:55:03.391724 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.391561    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p" (OuterVolumeSpecName: "kube-api-access-4zh9p") pod "de72eadc-12b7-4955-aac6-cdac6e594678" (UID: "de72eadc-12b7-4955-aac6-cdac6e594678"). InnerVolumeSpecName "kube-api-access-4zh9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:55:03.490437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490395    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490426    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-trusted-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490435    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de72eadc-12b7-4955-aac6-cdac6e594678-console-oauth-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490437 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490445    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-service-ca\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490454    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-oauth-serving-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490463    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zh9p\" (UniqueName: \"kubernetes.io/projected/de72eadc-12b7-4955-aac6-cdac6e594678-kube-api-access-4zh9p\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.490747 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.490471    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de72eadc-12b7-4955-aac6-cdac6e594678-console-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:55:03.963416 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:03.963384    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-kbz4f"
Jul 31 14:55:04.047483 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.047442    2575 generic.go:356] "Generic (PLEG): container finished" podID="de72eadc-12b7-4955-aac6-cdac6e594678" containerID="cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d" exitCode=0
Jul 31 14:55:04.047720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.047505    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d5d458b8-hjwz9"
Jul 31 14:55:04.047720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.047502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5d458b8-hjwz9" event={"ID":"de72eadc-12b7-4955-aac6-cdac6e594678","Type":"ContainerDied","Data":"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"}
Jul 31 14:55:04.047720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.047571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d5d458b8-hjwz9" event={"ID":"de72eadc-12b7-4955-aac6-cdac6e594678","Type":"ContainerDied","Data":"bf9ae5f75d65a556bd48e4b123ec9dc43990d47d72225e2c59bf98e5285416e8"}
Jul 31 14:55:04.047720 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.047624    2575 scope.go:117] "RemoveContainer" containerID="cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"
Jul 31 14:55:04.057118 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.057100    2575 scope.go:117] "RemoveContainer" containerID="cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"
Jul 31 14:55:04.057439 ip-10-0-143-227 kubenswrapper[2575]: E0731 14:55:04.057416    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d\": container with ID starting with cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d not found: ID does not exist" containerID="cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"
Jul 31 14:55:04.057513 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.057453    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d"} err="failed to get container status \"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d\": rpc error: code = NotFound desc = could not find container \"cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d\": container with ID starting with cbbfeb6c791a1716d9dd8f7ce19bd2112024555e45356214a328086bafd1df9d not found: ID does not exist"
Jul 31 14:55:04.075169 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.075129    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:55:04.079471 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:04.079438    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64d5d458b8-hjwz9"]
Jul 31 14:55:05.201417 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:05.201382    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de72eadc-12b7-4955-aac6-cdac6e594678" path="/var/lib/kubelet/pods/de72eadc-12b7-4955-aac6-cdac6e594678/volumes"
Jul 31 14:55:36.266042 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:36.266003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:55:36.403101 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:36.403068    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:55:46.940174 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:46.940134    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:55:47.045433 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:55:47.045402    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.200018 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.199982    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.304470 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.304431    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-swqdt"]
Jul 31 14:56:08.304797 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.304781    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de72eadc-12b7-4955-aac6-cdac6e594678" containerName="console"
Jul 31 14:56:08.304891 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.304800    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="de72eadc-12b7-4955-aac6-cdac6e594678" containerName="console"
Jul 31 14:56:08.304949 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.304888    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="de72eadc-12b7-4955-aac6-cdac6e594678" containerName="console"
Jul 31 14:56:08.308945 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.308926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.311732 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.311716    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:56:08.312515 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.312497    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 31 14:56:08.312589 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.312532    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:56:08.312850 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.312836    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-pxmcs\""
Jul 31 14:56:08.315648 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.315630    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.323139 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.323116    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-swqdt"]
Jul 31 14:56:08.353869 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.353843    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6pgxz"]
Jul 31 14:56:08.356774 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.356760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.359417 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.359400    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 14:56:08.359515 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.359447    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-p9lsd\""
Jul 31 14:56:08.369144 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.369125    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6pgxz"]
Jul 31 14:56:08.403588 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.403559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f42d126c-ac98-4e13-8815-00a84765897b-data\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.403588 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.403591    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnnd\" (UniqueName: \"kubernetes.io/projected/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-kube-api-access-jtnnd\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.403769 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.403639    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-cert\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.403769 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.403690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zcz\" (UniqueName: \"kubernetes.io/projected/f42d126c-ac98-4e13-8815-00a84765897b-kube-api-access-c5zcz\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.431852 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.431822    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.504896 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.504806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f42d126c-ac98-4e13-8815-00a84765897b-data\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.504896 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.504855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnd\" (UniqueName: \"kubernetes.io/projected/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-kube-api-access-jtnnd\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.505077 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.504974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-cert\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.505077 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.505002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zcz\" (UniqueName: \"kubernetes.io/projected/f42d126c-ac98-4e13-8815-00a84765897b-kube-api-access-c5zcz\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.505263 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.505243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f42d126c-ac98-4e13-8815-00a84765897b-data\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.507577 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.507557    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-cert\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.514798 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.514777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnnd\" (UniqueName: \"kubernetes.io/projected/5c13c4ca-b802-4903-9bf0-25935c3fb9cf-kube-api-access-jtnnd\") pod \"llmisvc-controller-manager-654c557856-swqdt\" (UID: \"5c13c4ca-b802-4903-9bf0-25935c3fb9cf\") " pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.514882 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.514831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zcz\" (UniqueName: \"kubernetes.io/projected/f42d126c-ac98-4e13-8815-00a84765897b-kube-api-access-c5zcz\") pod \"seaweedfs-64568bcd49-6pgxz\" (UID: \"f42d126c-ac98-4e13-8815-00a84765897b\") " pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.528210 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.528187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.619492 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.619467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:08.665592 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.665563    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:08.714107 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.713961    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:08.774451 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.774417    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-swqdt"]
Jul 31 14:56:08.778279 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:56:08.778240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c13c4ca_b802_4903_9bf0_25935c3fb9cf.slice/crio-1176ebd16ac39cc746e4e4032c60d9b15f9f1931c454d3e4fa59fa6fc695625b WatchSource:0}: Error finding container 1176ebd16ac39cc746e4e4032c60d9b15f9f1931c454d3e4fa59fa6fc695625b: Status 404 returned error can't find the container with id 1176ebd16ac39cc746e4e4032c60d9b15f9f1931c454d3e4fa59fa6fc695625b
Jul 31 14:56:08.819401 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:08.819367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6pgxz"]
Jul 31 14:56:08.822212 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:56:08.822189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42d126c_ac98_4e13_8815_00a84765897b.slice/crio-d2f8d5afa2f25851133f8ad649dc31422eff08e5f6898c9d331d251c7a995ce5 WatchSource:0}: Error finding container d2f8d5afa2f25851133f8ad649dc31422eff08e5f6898c9d331d251c7a995ce5: Status 404 returned error can't find the container with id d2f8d5afa2f25851133f8ad649dc31422eff08e5f6898c9d331d251c7a995ce5
Jul 31 14:56:09.061373 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:09.061291    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:09.199259 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:09.199228    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:56:09.329022 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:09.328932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-swqdt" event={"ID":"5c13c4ca-b802-4903-9bf0-25935c3fb9cf","Type":"ContainerStarted","Data":"1176ebd16ac39cc746e4e4032c60d9b15f9f1931c454d3e4fa59fa6fc695625b"}
Jul 31 14:56:09.330361 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:09.330310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6pgxz" event={"ID":"f42d126c-ac98-4e13-8815-00a84765897b","Type":"ContainerStarted","Data":"d2f8d5afa2f25851133f8ad649dc31422eff08e5f6898c9d331d251c7a995ce5"}
Jul 31 14:56:12.347281 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.347247    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-swqdt" event={"ID":"5c13c4ca-b802-4903-9bf0-25935c3fb9cf","Type":"ContainerStarted","Data":"36b164706c3a686c11700129cd1ca3d83afbe5051801bc8f52895a43417e6167"}
Jul 31 14:56:12.347762 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.347397    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:12.348930 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.348903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6pgxz" event={"ID":"f42d126c-ac98-4e13-8815-00a84765897b","Type":"ContainerStarted","Data":"0c9510f65b62f248570c531138ace9055ac63c84c48c753713810400c3911c99"}
Jul 31 14:56:12.349096 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.349041    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:12.367388 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.367322    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-swqdt" podStartSLOduration=1.062917338 podStartE2EDuration="4.367305987s" totalImagesPullingTime="3.304388649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:12.36579805 +0000 UTC m=+495.699917974" watchObservedRunningTime="2026-07-31 14:56:12.367305987 +0000 UTC m=+495.701425922"
Jul 31 14:56:12.387724 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:12.387668    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6pgxz" podStartSLOduration=1.076876869 podStartE2EDuration="4.387644243s" totalImagesPullingTime="3.310767374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:12.385997374 +0000 UTC m=+495.720117300" watchObservedRunningTime="2026-07-31 14:56:12.387644243 +0000 UTC m=+495.721764204"
Jul 31 14:56:18.354756 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:18.354719    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6pgxz"
Jul 31 14:56:43.355934 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:43.355902    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-swqdt"
Jul 31 14:56:54.660286 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:56:54.660194    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:29.976687 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:29.976650    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:30.059234 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.059197    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-s7pdd"]
Jul 31 14:57:30.061408 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.061389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.064284 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.064259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5vr64\""
Jul 31 14:57:30.065500 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.065477    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 31 14:57:30.065696 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.065478    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Jul 31 14:57:30.088922 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.088883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-s7pdd"]
Jul 31 14:57:30.128202 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.128166    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:30.167794 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.167759    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-cert\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.167992 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.167817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-metrics-certs\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.167992 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.167897    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94sd\" (UniqueName: \"kubernetes.io/projected/91f69b60-00f0-46c9-ba94-3f7989788237-kube-api-access-m94sd\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.231373 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.231285    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:30.269159 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.269123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-cert\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.269283 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.269177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-metrics-certs\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.269334 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.269291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94sd\" (UniqueName: \"kubernetes.io/projected/91f69b60-00f0-46c9-ba94-3f7989788237-kube-api-access-m94sd\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.271932 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.271908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-metrics-certs\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.271932 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.271925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f69b60-00f0-46c9-ba94-3f7989788237-cert\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.279735 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.279705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94sd\" (UniqueName: \"kubernetes.io/projected/91f69b60-00f0-46c9-ba94-3f7989788237-kube-api-access-m94sd\") pod \"odh-model-controller-8588677cfb-s7pdd\" (UID: \"91f69b60-00f0-46c9-ba94-3f7989788237\") " pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.373240 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.373196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:30.521233 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.521144    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-s7pdd"]
Jul 31 14:57:30.525968 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:57:30.525937    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f69b60_00f0_46c9_ba94_3f7989788237.slice/crio-7916a6639c20c24e7dea3f540ff1e7a57ff5c15ad345b64ca17384bd0310b441 WatchSource:0}: Error finding container 7916a6639c20c24e7dea3f540ff1e7a57ff5c15ad345b64ca17384bd0310b441: Status 404 returned error can't find the container with id 7916a6639c20c24e7dea3f540ff1e7a57ff5c15ad345b64ca17384bd0310b441
Jul 31 14:57:30.693035 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.693003    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-s7pdd" event={"ID":"91f69b60-00f0-46c9-ba94-3f7989788237","Type":"ContainerStarted","Data":"7916a6639c20c24e7dea3f540ff1e7a57ff5c15ad345b64ca17384bd0310b441"}
Jul 31 14:57:30.867446 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.867399    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:30.926619 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:30.926579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:57:33.709649 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:33.709586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-s7pdd" event={"ID":"91f69b60-00f0-46c9-ba94-3f7989788237","Type":"ContainerStarted","Data":"07a51c117dcb55e504a182664e43566068e2436491963d43917c484292fa225a"}
Jul 31 14:57:33.731092 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:33.731037    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-s7pdd" podStartSLOduration=1.276923385 podStartE2EDuration="3.731018999s" totalImagesPullingTime="2.454095614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:33.728500073 +0000 UTC m=+577.062619977" watchObservedRunningTime="2026-07-31 14:57:33.731018999 +0000 UTC m=+577.065138900"
Jul 31 14:57:34.713625 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:34.713575    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:45.720291 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:45.720258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-s7pdd"
Jul 31 14:57:46.641249 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.641210    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-69l7z"]
Jul 31 14:57:46.644801 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.644777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.666977 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.666934    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-69l7z"]
Jul 31 14:57:46.710577 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.710541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.710577 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.710581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcg7h\" (UniqueName: \"kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.811093 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.811056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.811093 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.811098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcg7h\" (UniqueName: \"kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.811715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.811569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.826145 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.826118    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcg7h\" (UniqueName: \"kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h\") pod \"s3-init-69l7z\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") " pod="kserve/s3-init-69l7z"
Jul 31 14:57:46.954382 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:46.954288    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-69l7z"
Jul 31 14:57:47.135071 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:47.135028    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-69l7z"]
Jul 31 14:57:47.141058 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:57:47.141028    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd029e8_7192_454e_9abf_5e04ba1f8060.slice/crio-62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0 WatchSource:0}: Error finding container 62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0: Status 404 returned error can't find the container with id 62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0
Jul 31 14:57:47.779394 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:47.779347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerStarted","Data":"62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0"}
Jul 31 14:57:50.799218 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:50.799170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerStarted","Data":"ce16b498f5263cc5170dbe28ccc31d9696f49180a37117f80ef54ca4c5d5034a"}
Jul 31 14:57:52.809608 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:52.809570    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerID="ce16b498f5263cc5170dbe28ccc31d9696f49180a37117f80ef54ca4c5d5034a" exitCode=0
Jul 31 14:57:52.809985 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:52.809665    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerDied","Data":"ce16b498f5263cc5170dbe28ccc31d9696f49180a37117f80ef54ca4c5d5034a"}
Jul 31 14:57:57.842924 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:57.842888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerStarted","Data":"17bb5614bb3cb3987cc09113d8e67fd15cfc110fc5a63e06cd4fd45aa68d6ea4"}
Jul 31 14:57:57.864273 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:57:57.864221    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-69l7z" podStartSLOduration=2.007170801 podStartE2EDuration="11.864202463s" totalImagesPullingTime="7.843326479s" totalInitContainerRuntime="2.013705183s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:57.862332373 +0000 UTC m=+601.196452296" watchObservedRunningTime="2026-07-31 14:57:57.864202463 +0000 UTC m=+601.198322384"
Jul 31 14:59:03.123144 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:03.123101    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerID="17bb5614bb3cb3987cc09113d8e67fd15cfc110fc5a63e06cd4fd45aa68d6ea4" exitCode=0
Jul 31 14:59:03.123574 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:03.123172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerDied","Data":"17bb5614bb3cb3987cc09113d8e67fd15cfc110fc5a63e06cd4fd45aa68d6ea4"}
Jul 31 14:59:04.261998 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.261960    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-69l7z"
Jul 31 14:59:04.355887 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.355843    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcg7h\" (UniqueName: \"kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h\") pod \"1dd029e8-7192-454e-9abf-5e04ba1f8060\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") "
Jul 31 14:59:04.356059 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.355947    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache\") pod \"1dd029e8-7192-454e-9abf-5e04ba1f8060\" (UID: \"1dd029e8-7192-454e-9abf-5e04ba1f8060\") "
Jul 31 14:59:04.358303 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.358266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h" (OuterVolumeSpecName: "kube-api-access-gcg7h") pod "1dd029e8-7192-454e-9abf-5e04ba1f8060" (UID: "1dd029e8-7192-454e-9abf-5e04ba1f8060"). InnerVolumeSpecName "kube-api-access-gcg7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:59:04.375695 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.375644    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd029e8-7192-454e-9abf-5e04ba1f8060" (UID: "1dd029e8-7192-454e-9abf-5e04ba1f8060"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:59:04.457312 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.457236    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcg7h\" (UniqueName: \"kubernetes.io/projected/1dd029e8-7192-454e-9abf-5e04ba1f8060-kube-api-access-gcg7h\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:59:04.457312 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:04.457269    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd029e8-7192-454e-9abf-5e04ba1f8060-model-cache\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 14:59:05.133729 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:05.133699    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-69l7z"
Jul 31 14:59:05.133905 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:05.133698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-69l7z" event={"ID":"1dd029e8-7192-454e-9abf-5e04ba1f8060","Type":"ContainerDied","Data":"62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0"}
Jul 31 14:59:05.133905 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:05.133812    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62a18d90dc1628210ecce8331cf4e83b7482b9ae424dc2da8b9b16481387aee0"
Jul 31 14:59:05.583415 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:05.583325    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:06.931613 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:06.931579    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:17.622637 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.622589    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:17.719972 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.719937    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:17.771245 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771197    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 14:59:17.771592 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771573    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerName="s3-init"
Jul 31 14:59:17.771592 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771594    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerName="s3-init"
Jul 31 14:59:17.771770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771648    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerName="download-hf-model"
Jul 31 14:59:17.771770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771657    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerName="download-hf-model"
Jul 31 14:59:17.771770 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.771742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd029e8-7192-454e-9abf-5e04ba1f8060" containerName="s3-init"
Jul 31 14:59:17.775151 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.775122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.778124 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.778052    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 14:59:17.778351 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.778258    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 14:59:17.778351 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.778307    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mhdbw\""
Jul 31 14:59:17.778530 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.778514    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 14:59:17.779441 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.779422    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 14:59:17.779564 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.779422    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-1a2d5-predictor-serving-cert\""
Jul 31 14:59:17.779564 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.779455    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-1a2d5-kube-rbac-proxy-sar-config\""
Jul 31 14:59:17.790854 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.790817    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 14:59:17.861518 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.861482    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:17.870697 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.870661    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97t9n\" (UniqueName: \"kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.870892 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.870725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.870892 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.870790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.870892 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.870855    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.870892 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.870875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972326 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97t9n\" (UniqueName: \"kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972326 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972326 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972656 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972656 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.972833 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.973008 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.972989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.973096 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.973078    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.975152 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.975126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:17.984444 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:17.984408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97t9n\" (UniqueName: \"kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n\") pod \"isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:18.003643 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.003591    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 14:59:18.006276 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.006247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.009262 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.009225    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 31 14:59:18.009578 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.009558    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 31 14:59:18.031439 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.031406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:18.073244 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.073204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.073244 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.073249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.073463 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.073284    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.073463 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.073338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.073463 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.073363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk664\" (UniqueName: \"kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.086688 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.086653    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:18.088167 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.088141    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 14:59:18.109779 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.109744    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 14:59:18.173978 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.173930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk664\" (UniqueName: \"kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174058    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174185 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174506 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.174799 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.175048 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.174994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.176903 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.176879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.182785 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.182512    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk664\" (UniqueName: \"kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664\") pod \"isvc-canary-rollback-predictor-7f985c86-xjtwh\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.240304 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.240207    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 14:59:18.244543 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:59:18.244511    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebdeff7_96aa_4345_b7d5_d61a8b925c44.slice/crio-43abe9c14900cb77436f04cb423210a8c10af0afac89596215952ecd3e7d510b WatchSource:0}: Error finding container 43abe9c14900cb77436f04cb423210a8c10af0afac89596215952ecd3e7d510b: Status 404 returned error can't find the container with id 43abe9c14900cb77436f04cb423210a8c10af0afac89596215952ecd3e7d510b
Jul 31 14:59:18.318241 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.318203    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:18.475654 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:18.475629    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 14:59:18.477866 ip-10-0-143-227 kubenswrapper[2575]: W0731 14:59:18.477836    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fa28d3_9097_4f90_bf49_5cf02fd3171f.slice/crio-64ad2812b89f39592938beffb3ada8876c807b8bd562712ee9106a05cfb7e6cf WatchSource:0}: Error finding container 64ad2812b89f39592938beffb3ada8876c807b8bd562712ee9106a05cfb7e6cf: Status 404 returned error can't find the container with id 64ad2812b89f39592938beffb3ada8876c807b8bd562712ee9106a05cfb7e6cf
Jul 31 14:59:19.203996 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:19.203954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerStarted","Data":"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"}
Jul 31 14:59:19.204511 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:19.204045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerStarted","Data":"43abe9c14900cb77436f04cb423210a8c10af0afac89596215952ecd3e7d510b"}
Jul 31 14:59:19.205710 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:19.205675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerStarted","Data":"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"}
Jul 31 14:59:19.205838 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:19.205734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerStarted","Data":"64ad2812b89f39592938beffb3ada8876c807b8bd562712ee9106a05cfb7e6cf"}
Jul 31 14:59:23.224727 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:23.224690    2575 generic.go:356] "Generic (PLEG): container finished" podID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerID="edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed" exitCode=0
Jul 31 14:59:23.225207 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:23.224765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerDied","Data":"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"}
Jul 31 14:59:23.226219 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:23.226199    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:59:23.226503 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:23.226484    2575 generic.go:356] "Generic (PLEG): container finished" podID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerID="71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e" exitCode=0
Jul 31 14:59:23.226625 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:23.226587    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerDied","Data":"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"}
Jul 31 14:59:40.374095 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:40.374052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerStarted","Data":"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"}
Jul 31 14:59:40.376649 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:40.376612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerStarted","Data":"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"}
Jul 31 14:59:43.396432 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:43.396388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerStarted","Data":"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"}
Jul 31 14:59:43.398910 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:43.398858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerStarted","Data":"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"}
Jul 31 14:59:43.399068 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:43.398984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:43.426322 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:43.426257    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podStartSLOduration=3.307580682 podStartE2EDuration="26.426235504s" totalImagesPullingTime="19.215978738s" totalInitContainerRuntime="3.902676084s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:43.424739842 +0000 UTC m=+706.758859766" watchObservedRunningTime="2026-07-31 14:59:43.426235504 +0000 UTC m=+706.760355491"
Jul 31 14:59:44.404647 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:44.404546    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:44.406322 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:44.406289    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 14:59:45.411285 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.411239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerStarted","Data":"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"}
Jul 31 14:59:45.411778 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.411361    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:45.411778 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.411609    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:45.411778 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.411584    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 14:59:45.413072 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.413043    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 14:59:45.440198 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:45.440138    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podStartSLOduration=3.348792219 podStartE2EDuration="28.440119957s" totalImagesPullingTime="21.094848746s" totalInitContainerRuntime="3.996478992s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:45.436575851 +0000 UTC m=+708.770695773" watchObservedRunningTime="2026-07-31 14:59:45.440119957 +0000 UTC m=+708.774239878"
Jul 31 14:59:46.416186 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:46.416152    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:46.416715 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:46.416307    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 14:59:46.417566 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:46.417538    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:47.421110 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:47.421057    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 14:59:47.421570 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:47.421532    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:47.425273 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:47.425247    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 14:59:48.426209 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:48.426101    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 14:59:48.426722 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:48.426650    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:50.416174 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:50.416144    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 14:59:50.416711 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:50.416687    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 14:59:58.426010 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:58.425964    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 14:59:58.426675 ip-10-0-143-227 kubenswrapper[2575]: I0731 14:59:58.426645    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:00.417434 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:00.417389    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:08.426438 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:08.426388    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:08.426940 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:08.426810    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:10.416756 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:10.416715    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:18.426846 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:18.426796    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:18.427347 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:18.427320    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:20.417452 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:20.417411    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:28.425977 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:28.425923    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:28.426672 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:28.426638    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:30.416863 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:30.416816    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:36.353006 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:36.352968    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:00:36.651021 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:36.650986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:00:38.426345 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:38.426290    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:38.426900 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:38.426869    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:40.416774 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:40.416732    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:48.426015 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:48.425955    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:48.426663 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:48.426627    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:49.297785 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:49.297752    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:00:50.416923 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:50.416880    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:55.197035 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:55.196990    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:00:58.425937 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:58.425886    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:00:58.426437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:58.426370    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:59.250054 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:00:59.250015    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:01.208125 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:01.208086    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:01.208541 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:01.208290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:05.201800 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:05.201764    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 15:01:07.730306 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:07.730269    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:12.797046 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:12.797007    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 15:01:12.797474 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:12.797284    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" containerID="cri-o://4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a" gracePeriod=30
Jul 31 15:01:12.797474 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:12.797363    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" containerID="cri-o://c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19" gracePeriod=30
Jul 31 15:01:12.797474 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:12.797404    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" containerID="cri-o://101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169" gracePeriod=30
Jul 31 15:01:12.828707 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:12.828663    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:13.024353 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.024314    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:01:13.027399 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.027378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.030026 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.029999    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b123a-predictor-serving-cert\""
Jul 31 15:01:13.030204 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.030002    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-b123a-kube-rbac-proxy-sar-config\""
Jul 31 15:01:13.037747 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.037715    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:01:13.110737 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.110705    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.110909 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.110748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.110909 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.110820    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjz4j\" (UniqueName: \"kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.110909 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.110887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.111016 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.110926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.189123 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.189080    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:13.212006 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.211957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212006 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212282 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212282 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjz4j\" (UniqueName: \"kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212282 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:13.212132    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-serving-cert: secret "isvc-xgboost-graph-raw-b123a-predictor-serving-cert" not found
Jul 31 15:01:13.212282 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212282 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:13.212283    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls podName:89ba51e5-3c2a-4bb3-bb76-a5496871bddf nodeName:}" failed. No retries permitted until 2026-07-31 15:01:13.712257266 +0000 UTC m=+797.046377183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls") pod "isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf") : secret "isvc-xgboost-graph-raw-b123a-predictor-serving-cert" not found
Jul 31 15:01:13.212546 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212521    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212817 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212800    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.212872 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.212802    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.224129 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.224095    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjz4j\" (UniqueName: \"kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.291055 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.291021    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:13.392145 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.392053    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:01:13.717192 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.717101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.719855 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.719826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") pod \"isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:13.840196 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.840159    2575 generic.go:356] "Generic (PLEG): container finished" podID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerID="101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169" exitCode=2
Jul 31 15:01:13.840652 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.840232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerDied","Data":"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"}
Jul 31 15:01:13.939880 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:13.939824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:14.089868 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:14.089837    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:01:14.092380 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:01:14.092352    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ba51e5_3c2a_4bb3_bb76_a5496871bddf.slice/crio-b1bdfafdbf5d10e9ffe2e0a037395d767bc0ca3cb855353d288ccb8f6d3e8184 WatchSource:0}: Error finding container b1bdfafdbf5d10e9ffe2e0a037395d767bc0ca3cb855353d288ccb8f6d3e8184: Status 404 returned error can't find the container with id b1bdfafdbf5d10e9ffe2e0a037395d767bc0ca3cb855353d288ccb8f6d3e8184
Jul 31 15:01:14.846428 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:14.846388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerStarted","Data":"fdadd6d707a965d3bf5bb25283229616266002cfd9e59ae71c6e2c619d6d1d82"}
Jul 31 15:01:14.846953 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:14.846468    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerStarted","Data":"b1bdfafdbf5d10e9ffe2e0a037395d767bc0ca3cb855353d288ccb8f6d3e8184"}
Jul 31 15:01:17.421377 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:17.421333    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:18.866715 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:18.866679    2575 generic.go:356] "Generic (PLEG): container finished" podID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerID="4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a" exitCode=0
Jul 31 15:01:18.867143 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:18.866754    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerDied","Data":"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"}
Jul 31 15:01:18.868333 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:18.868307    2575 generic.go:356] "Generic (PLEG): container finished" podID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerID="fdadd6d707a965d3bf5bb25283229616266002cfd9e59ae71c6e2c619d6d1d82" exitCode=0
Jul 31 15:01:18.868474 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:18.868380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerDied","Data":"fdadd6d707a965d3bf5bb25283229616266002cfd9e59ae71c6e2c619d6d1d82"}
Jul 31 15:01:21.196869 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:21.196773    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:01:21.197463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:21.197432    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:22.421266 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:22.421223    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:27.421667 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:27.421575    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:27.422118 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:27.421748    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:31.196936 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:31.196889    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:01:31.197510 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:31.197260    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:32.421494 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:32.421446    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:37.421797 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:37.421745    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:38.978301 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:38.978256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerStarted","Data":"d5ecf76024372d224213e3b7af7c08d2c100a6b9ffb691c4881b13b1d3656bd7"}
Jul 31 15:01:38.978935 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:38.978334    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerStarted","Data":"f331e6e1f6530cd1140f38a0d6fd8a54f0c902888548cd60ff0298ae8b69164e"}
Jul 31 15:01:38.978935 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:38.978523    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:39.003230 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:39.003164    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podStartSLOduration=3.105152962 podStartE2EDuration="26.003145526s" totalImagesPullingTime="19.132826486s" totalInitContainerRuntime="3.765166078s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:39.000706722 +0000 UTC m=+822.334826670" watchObservedRunningTime="2026-07-31 15:01:39.003145526 +0000 UTC m=+822.337265448"
Jul 31 15:01:39.983028 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:39.982992    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:39.984403 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:39.984367    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:01:40.987406 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:40.987364    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:01:41.197899 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:41.197837    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:01:41.198249 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:41.198226    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:41.202225 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:41.202202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:41.202340 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:41.202290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:42.421466 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:42.421422    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:01:43.465297 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.465266    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:43.583794 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.583679    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97t9n\" (UniqueName: \"kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n\") pod \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") "
Jul 31 15:01:43.583794 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.583754    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location\") pod \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") "
Jul 31 15:01:43.584040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.583802    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls\") pod \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") "
Jul 31 15:01:43.584040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.583831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config\") pod \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") "
Jul 31 15:01:43.584040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.583884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert\") pod \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\" (UID: \"1ebdeff7-96aa-4345-b7d5-d61a8b925c44\") "
Jul 31 15:01:43.584234 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.584166    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebdeff7-96aa-4345-b7d5-d61a8b925c44" (UID: "1ebdeff7-96aa-4345-b7d5-d61a8b925c44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:01:43.584299 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.584266    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1ebdeff7-96aa-4345-b7d5-d61a8b925c44" (UID: "1ebdeff7-96aa-4345-b7d5-d61a8b925c44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:01:43.584366 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.584341    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ebdeff7-96aa-4345-b7d5-d61a8b925c44" (UID: "1ebdeff7-96aa-4345-b7d5-d61a8b925c44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:01:43.586145 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.586121    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ebdeff7-96aa-4345-b7d5-d61a8b925c44" (UID: "1ebdeff7-96aa-4345-b7d5-d61a8b925c44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:01:43.586218 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.586161    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n" (OuterVolumeSpecName: "kube-api-access-97t9n") pod "1ebdeff7-96aa-4345-b7d5-d61a8b925c44" (UID: "1ebdeff7-96aa-4345-b7d5-d61a8b925c44"). InnerVolumeSpecName "kube-api-access-97t9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:01:43.684793 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.684735    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97t9n\" (UniqueName: \"kubernetes.io/projected/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-api-access-97t9n\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:01:43.684793 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.684779    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:01:43.684793 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.684789    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:01:43.684793 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.684800    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:01:43.684793 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:43.684808    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ebdeff7-96aa-4345-b7d5-d61a8b925c44-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:01:44.005376 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.005331    2575 generic.go:356] "Generic (PLEG): container finished" podID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerID="c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19" exitCode=0
Jul 31 15:01:44.005556 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.005410    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"
Jul 31 15:01:44.005556 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.005420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerDied","Data":"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"}
Jul 31 15:01:44.005556 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.005490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn" event={"ID":"1ebdeff7-96aa-4345-b7d5-d61a8b925c44","Type":"ContainerDied","Data":"43abe9c14900cb77436f04cb423210a8c10af0afac89596215952ecd3e7d510b"}
Jul 31 15:01:44.005556 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.005527    2575 scope.go:117] "RemoveContainer" containerID="c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"
Jul 31 15:01:44.014770 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.014751    2575 scope.go:117] "RemoveContainer" containerID="101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"
Jul 31 15:01:44.023395 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.023374    2575 scope.go:117] "RemoveContainer" containerID="4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"
Jul 31 15:01:44.030932 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.030900    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 15:01:44.032654 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.032625    2575 scope.go:117] "RemoveContainer" containerID="edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"
Jul 31 15:01:44.035138 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.035113    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-1a2d5-predictor-79d6cd6f99-jj6vn"]
Jul 31 15:01:44.041705 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.041681    2575 scope.go:117] "RemoveContainer" containerID="c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"
Jul 31 15:01:44.042060 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:44.042038    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19\": container with ID starting with c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19 not found: ID does not exist" containerID="c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"
Jul 31 15:01:44.042132 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042068    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19"} err="failed to get container status \"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19\": rpc error: code = NotFound desc = could not find container \"c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19\": container with ID starting with c51d788239912e70ba350965eabe6d7b473f83f3fd8ac23ce10e8532c9083c19 not found: ID does not exist"
Jul 31 15:01:44.042132 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042087    2575 scope.go:117] "RemoveContainer" containerID="101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"
Jul 31 15:01:44.042381 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:44.042363    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169\": container with ID starting with 101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169 not found: ID does not exist" containerID="101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"
Jul 31 15:01:44.042428 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042389    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169"} err="failed to get container status \"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169\": rpc error: code = NotFound desc = could not find container \"101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169\": container with ID starting with 101dc85d005545b8ece4cb0f91affc9c7fed32c578051df6249a63b1114b9169 not found: ID does not exist"
Jul 31 15:01:44.042428 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042405    2575 scope.go:117] "RemoveContainer" containerID="4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"
Jul 31 15:01:44.042652 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:44.042634    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a\": container with ID starting with 4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a not found: ID does not exist" containerID="4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"
Jul 31 15:01:44.042700 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042657    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a"} err="failed to get container status \"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a\": rpc error: code = NotFound desc = could not find container \"4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a\": container with ID starting with 4a6411b563bb4789e97355f123335037682d4808f885a5b109dcc344f8dbe85a not found: ID does not exist"
Jul 31 15:01:44.042700 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042670    2575 scope.go:117] "RemoveContainer" containerID="edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"
Jul 31 15:01:44.042902 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:01:44.042883    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed\": container with ID starting with edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed not found: ID does not exist" containerID="edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"
Jul 31 15:01:44.042950 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:44.042908    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed"} err="failed to get container status \"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed\": rpc error: code = NotFound desc = could not find container \"edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed\": container with ID starting with edf3e24fadfd733b4497c7e61a8c3bbce05bea6220feb6d520f05376a2b8aaed not found: ID does not exist"
Jul 31 15:01:45.202325 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:45.202290    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" path="/var/lib/kubelet/pods/1ebdeff7-96aa-4345-b7d5-d61a8b925c44/volumes"
Jul 31 15:01:45.991759 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:45.991727    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:01:45.992460 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:45.992431    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:01:55.993377 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:01:55.993329    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:02:05.993390 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:05.993344    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:02:15.993124 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:15.993071    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:02:25.993394 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:25.993352    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:02:35.992874 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:35.992819    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:02:37.740193 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:37.740150    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 15:02:37.740646 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:37.740547    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container" containerID="cri-o://ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f" gracePeriod=30
Jul 31 15:02:37.740646 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:37.740575    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kube-rbac-proxy" containerID="cri-o://bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3" gracePeriod=30
Jul 31 15:02:37.765763 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:37.765724    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:38.272358 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:38.272312    2575 generic.go:356] "Generic (PLEG): container finished" podID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerID="bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3" exitCode=2
Jul 31 15:02:38.272549 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:38.272381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerDied","Data":"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"}
Jul 31 15:02:40.412525 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:40.412473    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 15:02:42.912239 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.912202    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:42.966980 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.966945    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:02:42.967291 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967279    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967292    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967312    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967317    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967328    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="storage-initializer"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967334    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="storage-initializer"
Jul 31 15:02:42.967343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967341    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container"
Jul 31 15:02:42.967520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967346    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container"
Jul 31 15:02:42.967520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967401    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kube-rbac-proxy"
Jul 31 15:02:42.967520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967410    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="agent"
Jul 31 15:02:42.967520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.967418    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebdeff7-96aa-4345-b7d5-d61a8b925c44" containerName="kserve-container"
Jul 31 15:02:42.969542 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.969523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:42.972189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.972158    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 31 15:02:42.972357 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.972204    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 31 15:02:42.981739 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.981702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:02:42.995705 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:42.995664    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:43.085786 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.085749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.085955 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.085800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.085955 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.085851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.085955 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.085875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kj9v\" (UniqueName: \"kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.085955 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.085895    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.110882 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.110852    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 15:02:43.186948 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.186849    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls\") pod \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") "
Jul 31 15:02:43.186948 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.186906    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk664\" (UniqueName: \"kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664\") pod \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") "
Jul 31 15:02:43.186948 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.186930    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location\") pod \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") "
Jul 31 15:02:43.187176 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.186963    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert\") pod \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") "
Jul 31 15:02:43.187176 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187062    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config\") pod \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\" (UID: \"f6fa28d3-9097-4f90-bf49-5cf02fd3171f\") "
Jul 31 15:02:43.187273 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.187273 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kj9v\" (UniqueName: \"kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.187368 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187303    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6fa28d3-9097-4f90-bf49-5cf02fd3171f" (UID: "f6fa28d3-9097-4f90-bf49-5cf02fd3171f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:43.187368 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187338    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6fa28d3-9097-4f90-bf49-5cf02fd3171f" (UID: "f6fa28d3-9097-4f90-bf49-5cf02fd3171f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:02:43.187469 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.187469 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187411    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6fa28d3-9097-4f90-bf49-5cf02fd3171f" (UID: "f6fa28d3-9097-4f90-bf49-5cf02fd3171f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:02:43.187580 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.187502    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:02:43.187580 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.187580 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.187580 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.187563    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls podName:6fe864da-7c43-4693-ac91-9e7d8f31a3d8 nodeName:}" failed. No retries permitted until 2026-07-31 15:02:43.687541818 +0000 UTC m=+887.021661721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls") pod "isvc-canary-stop-predictor-6787f74994-bptm8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:02:43.187831 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187668    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.187831 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187687    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.187831 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187702    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.187991 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.187896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.188146 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.188122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.188214 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.188127    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.189992 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.189968    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6fa28d3-9097-4f90-bf49-5cf02fd3171f" (UID: "f6fa28d3-9097-4f90-bf49-5cf02fd3171f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:02:43.189992 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.189979    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664" (OuterVolumeSpecName: "kube-api-access-kk664") pod "f6fa28d3-9097-4f90-bf49-5cf02fd3171f" (UID: "f6fa28d3-9097-4f90-bf49-5cf02fd3171f"). InnerVolumeSpecName "kube-api-access-kk664". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:02:43.196345 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.196316    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kj9v\" (UniqueName: \"kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.289004 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.288963    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.289004 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.288995    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk664\" (UniqueName: \"kubernetes.io/projected/f6fa28d3-9097-4f90-bf49-5cf02fd3171f-kube-api-access-kk664\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.296936 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.296901    2575 generic.go:356] "Generic (PLEG): container finished" podID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerID="ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f" exitCode=0
Jul 31 15:02:43.297112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.296966    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"
Jul 31 15:02:43.297112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.296982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerDied","Data":"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"}
Jul 31 15:02:43.297112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.297055    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh" event={"ID":"f6fa28d3-9097-4f90-bf49-5cf02fd3171f","Type":"ContainerDied","Data":"64ad2812b89f39592938beffb3ada8876c807b8bd562712ee9106a05cfb7e6cf"}
Jul 31 15:02:43.297112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.297093    2575 scope.go:117] "RemoveContainer" containerID="bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"
Jul 31 15:02:43.307532 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.306723    2575 scope.go:117] "RemoveContainer" containerID="ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"
Jul 31 15:02:43.318562 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.318522    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 15:02:43.320123 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.320099    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-xjtwh"]
Jul 31 15:02:43.321623 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.321566    2575 scope.go:117] "RemoveContainer" containerID="71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"
Jul 31 15:02:43.330433 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.330408    2575 scope.go:117] "RemoveContainer" containerID="bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"
Jul 31 15:02:43.330780 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.330757    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3\": container with ID starting with bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3 not found: ID does not exist" containerID="bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"
Jul 31 15:02:43.330841 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.330791    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3"} err="failed to get container status \"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3\": rpc error: code = NotFound desc = could not find container \"bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3\": container with ID starting with bfc0eb251675672f61a64be88631615bbec330058eb2f02bc38d90231ef37bf3 not found: ID does not exist"
Jul 31 15:02:43.330841 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.330811    2575 scope.go:117] "RemoveContainer" containerID="ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"
Jul 31 15:02:43.331084 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.331064    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f\": container with ID starting with ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f not found: ID does not exist" containerID="ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"
Jul 31 15:02:43.331154 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.331095    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f"} err="failed to get container status \"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f\": rpc error: code = NotFound desc = could not find container \"ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f\": container with ID starting with ca70590241e75c501387d92a325b0756deab52d9232dc6654addfbcd1075590f not found: ID does not exist"
Jul 31 15:02:43.331154 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.331117    2575 scope.go:117] "RemoveContainer" containerID="71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"
Jul 31 15:02:43.331405 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.331388    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e\": container with ID starting with 71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e not found: ID does not exist" containerID="71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"
Jul 31 15:02:43.331453 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.331408    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e"} err="failed to get container status \"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e\": rpc error: code = NotFound desc = could not find container \"71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e\": container with ID starting with 71ae369ddb8fb35b99388be2b39f205c37ef43e79a50acc77e650268156ddc2e not found: ID does not exist"
Jul 31 15:02:43.350174 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.350133    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:43.693383 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:43.693339    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:43.693659 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.693537    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:02:43.693744 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:02:43.693671    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls podName:6fe864da-7c43-4693-ac91-9e7d8f31a3d8 nodeName:}" failed. No retries permitted until 2026-07-31 15:02:44.693644827 +0000 UTC m=+888.027764745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls") pod "isvc-canary-stop-predictor-6787f74994-bptm8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:02:44.113234 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:44.113184    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:44.702870 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:44.702824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:44.705687 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:44.705654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-bptm8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:44.832017 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:44.831982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:44.981297 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:44.981269    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:02:44.983899 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:02:44.983871    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe864da_7c43_4693_ac91_9e7d8f31a3d8.slice/crio-e504b606604a2be2b34391c7f5d2db261fb497a2cf3374511e282ccb76a835ce WatchSource:0}: Error finding container e504b606604a2be2b34391c7f5d2db261fb497a2cf3374511e282ccb76a835ce: Status 404 returned error can't find the container with id e504b606604a2be2b34391c7f5d2db261fb497a2cf3374511e282ccb76a835ce
Jul 31 15:02:45.202724 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:45.202682    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" path="/var/lib/kubelet/pods/f6fa28d3-9097-4f90-bf49-5cf02fd3171f/volumes"
Jul 31 15:02:45.310217 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:45.310117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerStarted","Data":"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"}
Jul 31 15:02:45.310217 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:45.310178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerStarted","Data":"e504b606604a2be2b34391c7f5d2db261fb497a2cf3374511e282ccb76a835ce"}
Jul 31 15:02:45.993538 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:45.993506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:02:49.330255 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:49.330165    2575 generic.go:356] "Generic (PLEG): container finished" podID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerID="2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234" exitCode=0
Jul 31 15:02:49.330255 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:49.330238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerDied","Data":"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"}
Jul 31 15:02:50.337705 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.337667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerStarted","Data":"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"}
Jul 31 15:02:50.338135 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.337736    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerStarted","Data":"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"}
Jul 31 15:02:50.338135 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.338009    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:50.338209 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.338165    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:50.339513 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.339481    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:02:50.363564 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:50.363497    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podStartSLOduration=4.54124059 podStartE2EDuration="8.363476576s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.822235986s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:50.361721681 +0000 UTC m=+893.695841604" watchObservedRunningTime="2026-07-31 15:02:50.363476576 +0000 UTC m=+893.697596500"
Jul 31 15:02:51.342806 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:51.342769    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:02:52.934799 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.934762    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935095    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="storage-initializer"
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935108    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="storage-initializer"
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935115    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container"
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935121    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container"
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935134    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kube-rbac-proxy"
Jul 31 15:02:52.935184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935139    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kube-rbac-proxy"
Jul 31 15:02:52.935373 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935197    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kube-rbac-proxy"
Jul 31 15:02:52.935373 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.935207    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6fa28d3-9097-4f90-bf49-5cf02fd3171f" containerName="kserve-container"
Jul 31 15:02:52.937961 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.937940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:52.943334 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.943309    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-b123a-kube-rbac-proxy-sar-config\""
Jul 31 15:02:52.943479 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.943309    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-b123a-serving-cert\""
Jul 31 15:02:52.952200 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.952166    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:02:52.954285 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.954252    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:52.974080 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.974042    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:52.974247 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:52.974184    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.050346 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.050308    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:02:53.075509 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.075471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.075653 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.075518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.076172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.076144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.078299 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.078273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls\") pod \"model-chainer-raw-b123a-b9cfdbffc-g78z8\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") " pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.249143 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.249037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:02:53.397896 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:53.397861    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:02:53.400166 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:02:53.400137    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2379696c_9017_4c68_b713_5741068a1d5e.slice/crio-34c3718f0a7224d895a1624f2afc7d6a9ac56392d82a89db0ed73225117e9d8e WatchSource:0}: Error finding container 34c3718f0a7224d895a1624f2afc7d6a9ac56392d82a89db0ed73225117e9d8e: Status 404 returned error can't find the container with id 34c3718f0a7224d895a1624f2afc7d6a9ac56392d82a89db0ed73225117e9d8e
Jul 31 15:02:54.358647 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:54.358585    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" event={"ID":"2379696c-9017-4c68-b713-5741068a1d5e","Type":"ContainerStarted","Data":"34c3718f0a7224d895a1624f2afc7d6a9ac56392d82a89db0ed73225117e9d8e"}
Jul 31 15:02:56.348112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:56.348084    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:02:56.348629 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:56.348586    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:02:56.373817 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:56.373784    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" event={"ID":"2379696c-9017-4c68-b713-5741068a1d5e","Type":"ContainerStarted","Data":"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"}
Jul 31 15:02:56.374036 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:02:56.374019    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:03:02.384152 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:02.384122    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:03:02.408815 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:02.408751    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podStartSLOduration=8.324367116 podStartE2EDuration="10.408732297s" totalImagesPullingTime="2.084365181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:56.391015205 +0000 UTC m=+899.725135124" watchObservedRunningTime="2026-07-31 15:03:02.408732297 +0000 UTC m=+905.742852218"
Jul 31 15:03:02.995465 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:02.995426    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:03:02.995690 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:02.995666    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" containerID="cri-o://eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7" gracePeriod=30
Jul 31 15:03:03.018626 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.018577    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:03.153034 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.153003    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:03.192308 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.192270    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:03:03.197502 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.197473    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.200192 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.200163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e936d-predictor-serving-cert\""
Jul 31 15:03:03.200352 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.200220    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e936d-kube-rbac-proxy-sar-config\""
Jul 31 15:03:03.208071 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.208036    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:03:03.242858 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.242824    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:03.278165 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.278062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.278165 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.278106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sttp8\" (UniqueName: \"kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.278165 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.278162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.278452 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.278188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.278452 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.278296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.280180 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.280153    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:03:03.280450 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.280422    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" containerID="cri-o://f331e6e1f6530cd1140f38a0d6fd8a54f0c902888548cd60ff0298ae8b69164e" gracePeriod=30
Jul 31 15:03:03.280539 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.280456    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kube-rbac-proxy" containerID="cri-o://d5ecf76024372d224213e3b7af7c08d2c100a6b9ffb691c4881b13b1d3656bd7" gracePeriod=30
Jul 31 15:03:03.311215 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.311180    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:03:03.315461 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.315432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.317870 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.317835    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e936d-predictor-serving-cert\""
Jul 31 15:03:03.318062 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.318046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e936d-kube-rbac-proxy-sar-config\""
Jul 31 15:03:03.332426 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.332362    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:03:03.343428 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.343395    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:03.379185 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379145    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.379185 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.379414 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.379414 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.379414 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sttp8\" (UniqueName: \"kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.379760 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.379734    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.380107 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.380086    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.380174 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.380143    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.382044 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.382026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.389658 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.389620    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sttp8\" (UniqueName: \"kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8\") pod \"isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.408529 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.408492    2575 generic.go:356] "Generic (PLEG): container finished" podID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerID="d5ecf76024372d224213e3b7af7c08d2c100a6b9ffb691c4881b13b1d3656bd7" exitCode=2
Jul 31 15:03:03.408866 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.408559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerDied","Data":"d5ecf76024372d224213e3b7af7c08d2c100a6b9ffb691c4881b13b1d3656bd7"}
Jul 31 15:03:03.434423 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.434387    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:03.480576 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.480540    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.480576 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.480580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.480825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.480658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.480825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.480706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6p7c\" (UniqueName: \"kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.480825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.480733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.509102 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.509062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:03.582229 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.582408 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.582408 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.582408 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p7c\" (UniqueName: \"kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.582408 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.582885 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.582864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.583038 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.583013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.583138 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.583104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.585212 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.585184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.591832 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.591790    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p7c\" (UniqueName: \"kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c\") pod \"isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.626109 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.626063    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:03.662292 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.662263    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:03:03.798453 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:03.798412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:03:03.803094 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:03:03.803066    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72ab21f_8fd4_4b76_ad0b_94a251a5d62f.slice/crio-ae0526a127f45273a0625bc90c0e0613b0de478e6a86e81c0a73de943fd5ccc8 WatchSource:0}: Error finding container ae0526a127f45273a0625bc90c0e0613b0de478e6a86e81c0a73de943fd5ccc8: Status 404 returned error can't find the container with id ae0526a127f45273a0625bc90c0e0613b0de478e6a86e81c0a73de943fd5ccc8
Jul 31 15:03:04.426457 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:04.426400    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerStarted","Data":"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"}
Jul 31 15:03:04.427005 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:04.426523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerStarted","Data":"a60a095804315808391ed2841a0eaac71f91cabb878bcccef311cca47f446bbe"}
Jul 31 15:03:04.428725 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:04.428683    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerStarted","Data":"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"}
Jul 31 15:03:04.428902 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:04.428757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerStarted","Data":"ae0526a127f45273a0625bc90c0e0613b0de478e6a86e81c0a73de943fd5ccc8"}
Jul 31 15:03:05.987565 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:05.987512    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 15:03:05.992932 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:05.992898    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:03:06.349454 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:06.349352    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:07.382359 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.382313    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:07.447778 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.447737    2575 generic.go:356] "Generic (PLEG): container finished" podID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerID="f331e6e1f6530cd1140f38a0d6fd8a54f0c902888548cd60ff0298ae8b69164e" exitCode=0
Jul 31 15:03:07.447778 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.447773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerDied","Data":"f331e6e1f6530cd1140f38a0d6fd8a54f0c902888548cd60ff0298ae8b69164e"}
Jul 31 15:03:07.547276 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.547238    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:03:07.718937 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.718898    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") pod \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") "
Jul 31 15:03:07.719139 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.718973    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config\") pod \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") "
Jul 31 15:03:07.719139 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719000    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert\") pod \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") "
Jul 31 15:03:07.719139 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719037    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location\") pod \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") "
Jul 31 15:03:07.719139 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719073    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjz4j\" (UniqueName: \"kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j\") pod \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\" (UID: \"89ba51e5-3c2a-4bb3-bb76-a5496871bddf\") "
Jul 31 15:03:07.719369 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719334    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89ba51e5-3c2a-4bb3-bb76-a5496871bddf" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:07.719369 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719338    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89ba51e5-3c2a-4bb3-bb76-a5496871bddf" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:03:07.719369 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.719364    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89ba51e5-3c2a-4bb3-bb76-a5496871bddf" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:07.721642 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.721588    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89ba51e5-3c2a-4bb3-bb76-a5496871bddf" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:07.721642 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.721590    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j" (OuterVolumeSpecName: "kube-api-access-sjz4j") pod "89ba51e5-3c2a-4bb3-bb76-a5496871bddf" (UID: "89ba51e5-3c2a-4bb3-bb76-a5496871bddf"). InnerVolumeSpecName "kube-api-access-sjz4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:07.819905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.819847    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:07.819905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.819897    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:07.819905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.819911    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:07.820156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.819924    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjz4j\" (UniqueName: \"kubernetes.io/projected/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-kube-api-access-sjz4j\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:07.820156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:07.819936    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89ba51e5-3c2a-4bb3-bb76-a5496871bddf-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:08.454371 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.454336    2575 generic.go:356] "Generic (PLEG): container finished" podID="93b19444-3315-44f0-8112-c1f1513f97ed" containerID="65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b" exitCode=0
Jul 31 15:03:08.454837 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.454411    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerDied","Data":"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"}
Jul 31 15:03:08.456164 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.456136    2575 generic.go:356] "Generic (PLEG): container finished" podID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerID="cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a" exitCode=0
Jul 31 15:03:08.456266 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.456167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerDied","Data":"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"}
Jul 31 15:03:08.458388 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.458366    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"
Jul 31 15:03:08.458388 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.458375    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr" event={"ID":"89ba51e5-3c2a-4bb3-bb76-a5496871bddf","Type":"ContainerDied","Data":"b1bdfafdbf5d10e9ffe2e0a037395d767bc0ca3cb855353d288ccb8f6d3e8184"}
Jul 31 15:03:08.458578 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.458451    2575 scope.go:117] "RemoveContainer" containerID="d5ecf76024372d224213e3b7af7c08d2c100a6b9ffb691c4881b13b1d3656bd7"
Jul 31 15:03:08.478331 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.478302    2575 scope.go:117] "RemoveContainer" containerID="f331e6e1f6530cd1140f38a0d6fd8a54f0c902888548cd60ff0298ae8b69164e"
Jul 31 15:03:08.489364 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.489328    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:03:08.494778 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.494738    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-b123a-predictor-75d58954c7-98zsr"]
Jul 31 15:03:08.500811 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:08.500761    2575 scope.go:117] "RemoveContainer" containerID="fdadd6d707a965d3bf5bb25283229616266002cfd9e59ae71c6e2c619d6d1d82"
Jul 31 15:03:09.203225 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.203182    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" path="/var/lib/kubelet/pods/89ba51e5-3c2a-4bb3-bb76-a5496871bddf/volumes"
Jul 31 15:03:09.466739 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.466629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerStarted","Data":"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"}
Jul 31 15:03:09.466739 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.466707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerStarted","Data":"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"}
Jul 31 15:03:09.467257 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.466918    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:09.468908 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.468881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerStarted","Data":"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"}
Jul 31 15:03:09.469044 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.468938    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerStarted","Data":"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"}
Jul 31 15:03:09.469199 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.469184    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:09.469245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.469225    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:09.470807 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.470775    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:09.490661 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.490584    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podStartSLOduration=2.64380841 podStartE2EDuration="6.490563814s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.846755404s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:09.487889591 +0000 UTC m=+912.822009513" watchObservedRunningTime="2026-07-31 15:03:09.490563814 +0000 UTC m=+912.824683736"
Jul 31 15:03:09.512189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:09.512137    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podStartSLOduration=2.7584808819999997 podStartE2EDuration="6.51212305s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.753642168s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:09.51199122 +0000 UTC m=+912.846111141" watchObservedRunningTime="2026-07-31 15:03:09.51212305 +0000 UTC m=+912.846242971"
Jul 31 15:03:10.474101 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:10.474056    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:10.474585 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:10.474231    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:10.475813 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:10.475782    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:11.479624 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:11.479550    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:12.382469 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:12.382430    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:15.478534 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:15.478504    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:03:15.479065 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:15.479037    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:16.349579 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:16.349535    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:16.484722 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:16.484693    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:03:16.485220 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:16.485195    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:17.382707 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:17.382666    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:17.382887 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:17.382773    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:03:22.382854 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:22.382806    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:25.479116 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:25.479068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:26.348825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:26.348784    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:26.485737 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:26.485691    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:27.383557 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:27.383516    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:28.541884 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:28.541846    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:03:32.382544 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:32.382491    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:33.161433 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.161404    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:03:33.241167 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.241134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle\") pod \"2379696c-9017-4c68-b713-5741068a1d5e\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") "
Jul 31 15:03:33.241364 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.241198    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls\") pod \"2379696c-9017-4c68-b713-5741068a1d5e\" (UID: \"2379696c-9017-4c68-b713-5741068a1d5e\") "
Jul 31 15:03:33.241550 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.241523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2379696c-9017-4c68-b713-5741068a1d5e" (UID: "2379696c-9017-4c68-b713-5741068a1d5e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:33.243462 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.243425    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2379696c-9017-4c68-b713-5741068a1d5e" (UID: "2379696c-9017-4c68-b713-5741068a1d5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:33.342407 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.342315    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2379696c-9017-4c68-b713-5741068a1d5e-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:33.342407 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.342348    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2379696c-9017-4c68-b713-5741068a1d5e-openshift-service-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:03:33.593203 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.593106    2575 generic.go:356] "Generic (PLEG): container finished" podID="2379696c-9017-4c68-b713-5741068a1d5e" containerID="eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7" exitCode=0
Jul 31 15:03:33.593203 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.593157    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"
Jul 31 15:03:33.593663 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.593208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" event={"ID":"2379696c-9017-4c68-b713-5741068a1d5e","Type":"ContainerDied","Data":"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"}
Jul 31 15:03:33.593663 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.593265    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8" event={"ID":"2379696c-9017-4c68-b713-5741068a1d5e","Type":"ContainerDied","Data":"34c3718f0a7224d895a1624f2afc7d6a9ac56392d82a89db0ed73225117e9d8e"}
Jul 31 15:03:33.593663 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.593287    2575 scope.go:117] "RemoveContainer" containerID="eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"
Jul 31 15:03:33.602960 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.602941    2575 scope.go:117] "RemoveContainer" containerID="eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"
Jul 31 15:03:33.603284 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:03:33.603266    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7\": container with ID starting with eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7 not found: ID does not exist" containerID="eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"
Jul 31 15:03:33.603350 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.603296    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7"} err="failed to get container status \"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7\": rpc error: code = NotFound desc = could not find container \"eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7\": container with ID starting with eaafa18ed88adb9371ea5124c08fa991509d179951cbbc1ecfca1cbeb23424f7 not found: ID does not exist"
Jul 31 15:03:33.617457 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.617420    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:03:33.623294 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:33.623248    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-b123a-b9cfdbffc-g78z8"]
Jul 31 15:03:35.202509 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:35.202469    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2379696c-9017-4c68-b713-5741068a1d5e" path="/var/lib/kubelet/pods/2379696c-9017-4c68-b713-5741068a1d5e/volumes"
Jul 31 15:03:35.479851 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:35.479752    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:36.349645 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:36.349577    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:36.485937 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:36.485892    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:45.479459 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:45.479410    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:46.349266 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:46.349219    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:46.486204 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:46.486162    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:03:55.480064 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:55.480021    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:03:56.349337 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:56.349288    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:03:56.485546 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:03:56.485495    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:04:05.480024 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:05.479976    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:04:06.348848 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:06.348799    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:04:06.486111 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:06.486053    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:04:08.197135 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:08.197094    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:04:15.480498 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:15.480458    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:04:16.485858 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:16.485818    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:04:18.197879 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:18.197790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:04:22.937984 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:22.937943    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:04:22.938575 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:22.938220    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" containerID="cri-o://9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865" gracePeriod=30
Jul 31 15:04:22.938575 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:22.938222    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kube-rbac-proxy" containerID="cri-o://7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b" gracePeriod=30
Jul 31 15:04:22.975155 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:22.975121    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:23.856827 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:23.856790    2575 generic.go:356] "Generic (PLEG): container finished" podID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerID="7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b" exitCode=2
Jul 31 15:04:23.857032 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:23.856872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerDied","Data":"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"}
Jul 31 15:04:24.104329 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:24.104293    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:26.343810 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:26.343762    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 31 15:04:26.485905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:26.485860    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:04:28.197571 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.197528    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:04:28.302361 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.302328    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:04:28.428660 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428526    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config\") pod \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") "
Jul 31 15:04:28.428660 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428629    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kj9v\" (UniqueName: \"kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v\") pod \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") "
Jul 31 15:04:28.428903 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428735    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") pod \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") "
Jul 31 15:04:28.428903 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428773    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert\") pod \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") "
Jul 31 15:04:28.428903 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428833    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location\") pod \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\" (UID: \"6fe864da-7c43-4693-ac91-9e7d8f31a3d8\") "
Jul 31 15:04:28.429050 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.428975    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fe864da-7c43-4693-ac91-9e7d8f31a3d8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:28.429187 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.429166    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fe864da-7c43-4693-ac91-9e7d8f31a3d8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:28.429257 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.429211    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fe864da-7c43-4693-ac91-9e7d8f31a3d8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:28.429315 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.429270    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:04:28.429315 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.429283    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:04:28.430971 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.430938    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fe864da-7c43-4693-ac91-9e7d8f31a3d8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:28.431102 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.430960    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v" (OuterVolumeSpecName: "kube-api-access-9kj9v") pod "6fe864da-7c43-4693-ac91-9e7d8f31a3d8" (UID: "6fe864da-7c43-4693-ac91-9e7d8f31a3d8"). InnerVolumeSpecName "kube-api-access-9kj9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:28.529713 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.529683    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kj9v\" (UniqueName: \"kubernetes.io/projected/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kube-api-access-9kj9v\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:04:28.529713 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.529709    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:04:28.529713 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.529719    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe864da-7c43-4693-ac91-9e7d8f31a3d8-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:04:28.881518 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.881483    2575 generic.go:356] "Generic (PLEG): container finished" podID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerID="9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865" exitCode=0
Jul 31 15:04:28.881767 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.881547    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"
Jul 31 15:04:28.881767 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.881569    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerDied","Data":"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"}
Jul 31 15:04:28.881767 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.881663    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8" event={"ID":"6fe864da-7c43-4693-ac91-9e7d8f31a3d8","Type":"ContainerDied","Data":"e504b606604a2be2b34391c7f5d2db261fb497a2cf3374511e282ccb76a835ce"}
Jul 31 15:04:28.881767 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.881702    2575 scope.go:117] "RemoveContainer" containerID="7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"
Jul 31 15:04:28.891191 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.891169    2575 scope.go:117] "RemoveContainer" containerID="9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"
Jul 31 15:04:28.900242 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.900217    2575 scope.go:117] "RemoveContainer" containerID="2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"
Jul 31 15:04:28.905331 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.905298    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:04:28.909336 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.909312    2575 scope.go:117] "RemoveContainer" containerID="7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"
Jul 31 15:04:28.909881 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:04:28.909843    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b\": container with ID starting with 7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b not found: ID does not exist" containerID="7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"
Jul 31 15:04:28.910014 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.909893    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b"} err="failed to get container status \"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b\": rpc error: code = NotFound desc = could not find container \"7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b\": container with ID starting with 7e2bb652936769228b7edd6de3504cc72f0ae763dacdf9d3c7ea5929fdfd7b8b not found: ID does not exist"
Jul 31 15:04:28.910014 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.909923    2575 scope.go:117] "RemoveContainer" containerID="9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"
Jul 31 15:04:28.910269 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:04:28.910250    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865\": container with ID starting with 9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865 not found: ID does not exist" containerID="9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"
Jul 31 15:04:28.910340 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.910277    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865"} err="failed to get container status \"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865\": rpc error: code = NotFound desc = could not find container \"9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865\": container with ID starting with 9e511b2ecd9653d574078cc1a42abcf870c45afa8fa2ebb56fd3758e456bb865 not found: ID does not exist"
Jul 31 15:04:28.910340 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.910296    2575 scope.go:117] "RemoveContainer" containerID="2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"
Jul 31 15:04:28.910546 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:04:28.910525    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234\": container with ID starting with 2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234 not found: ID does not exist" containerID="2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"
Jul 31 15:04:28.910634 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.910556    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234"} err="failed to get container status \"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234\": rpc error: code = NotFound desc = could not find container \"2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234\": container with ID starting with 2e9f4e550074b8035c0462f7338cf60fbdf40aaf3daa48b90051dfc626d11234 not found: ID does not exist"
Jul 31 15:04:28.911544 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:28.911525    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-bptm8"]
Jul 31 15:04:29.202674 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:29.202564    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" path="/var/lib/kubelet/pods/6fe864da-7c43-4693-ac91-9e7d8f31a3d8/volumes"
Jul 31 15:04:33.190870 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:33.190835    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:33.370321 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:33.370267    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:36.486378 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:36.486345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:04:53.241437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241385    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241919    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241940    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241957    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241965    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241979    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="storage-initializer"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.241988    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="storage-initializer"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242002    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242012    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242029    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242038    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242052    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242059    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242073    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="storage-initializer"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242082    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="storage-initializer"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242159    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242172    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe864da-7c43-4693-ac91-9e7d8f31a3d8" containerName="kserve-container"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242182    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kserve-container"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242190    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ba51e5-3c2a-4bb3-bb76-a5496871bddf" containerName="kube-rbac-proxy"
Jul 31 15:04:53.242734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.242202    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2379696c-9017-4c68-b713-5741068a1d5e" containerName="model-chainer-raw-b123a"
Jul 31 15:04:53.245712 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.245682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.248312 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.248283    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e936d-serving-cert\""
Jul 31 15:04:53.248464 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.248283    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e936d-kube-rbac-proxy-sar-config\""
Jul 31 15:04:53.253508 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.253102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:04:53.257840 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.257806    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:53.356245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.356208    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.356245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.356258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.457665 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.457582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.457829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.457748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.457876 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:04:53.457866    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-serving-cert: secret "model-chainer-raw-hpa-e936d-serving-cert" not found
Jul 31 15:04:53.457972 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:04:53.457959    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls podName:4320da7a-a254-4d03-87a3-f9ff9115fc55 nodeName:}" failed. No retries permitted until 2026-07-31 15:04:53.957933356 +0000 UTC m=+1017.292053255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls") pod "model-chainer-raw-hpa-e936d-84649d8867-c2cdj" (UID: "4320da7a-a254-4d03-87a3-f9ff9115fc55") : secret "model-chainer-raw-hpa-e936d-serving-cert" not found
Jul 31 15:04:53.458394 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.458374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.865994 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.865948    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:04:53.962037 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.962006    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:53.964711 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:53.964675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") pod \"model-chainer-raw-hpa-e936d-84649d8867-c2cdj\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:54.157152 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:54.157048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:54.309438 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:54.306508    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:04:54.310224 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:04:54.310193    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4320da7a_a254_4d03_87a3_f9ff9115fc55.slice/crio-fb279b140a59d078a3fab57ee71ad13e050c948e558fef4273490f723a365414 WatchSource:0}: Error finding container fb279b140a59d078a3fab57ee71ad13e050c948e558fef4273490f723a365414: Status 404 returned error can't find the container with id fb279b140a59d078a3fab57ee71ad13e050c948e558fef4273490f723a365414
Jul 31 15:04:55.011328 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:55.011288    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" event={"ID":"4320da7a-a254-4d03-87a3-f9ff9115fc55","Type":"ContainerStarted","Data":"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"}
Jul 31 15:04:55.011542 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:55.011346    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" event={"ID":"4320da7a-a254-4d03-87a3-f9ff9115fc55","Type":"ContainerStarted","Data":"fb279b140a59d078a3fab57ee71ad13e050c948e558fef4273490f723a365414"}
Jul 31 15:04:55.011542 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:55.011407    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:04:55.029005 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:04:55.028940    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podStartSLOduration=2.028921109 podStartE2EDuration="2.028921109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:55.028374686 +0000 UTC m=+1018.362494808" watchObservedRunningTime="2026-07-31 15:04:55.028921109 +0000 UTC m=+1018.363041030"
Jul 31 15:05:01.022467 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:01.022436    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:05:03.300748 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.300702    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:05:03.301144 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.300934    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" containerID="cri-o://b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08" gracePeriod=30
Jul 31 15:05:03.332069 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.332030    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:03.437402 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.437356    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:05:03.437758 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.437710    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" containerID="cri-o://ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f" gracePeriod=30
Jul 31 15:05:03.437758 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.437742    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kube-rbac-proxy" containerID="cri-o://c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5" gracePeriod=30
Jul 31 15:05:03.465535 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.465487    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:03.548933 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.548903    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:03.568115 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.568076    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:05:03.568441 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.568400    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" containerID="cri-o://2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8" gracePeriod=30
Jul 31 15:05:03.568567 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.568449    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kube-rbac-proxy" containerID="cri-o://f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5" gracePeriod=30
Jul 31 15:05:03.653826 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.653792    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:03.943481 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:03.943443    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:04.061409 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:04.061372    2575 generic.go:356] "Generic (PLEG): container finished" podID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerID="f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5" exitCode=2
Jul 31 15:05:04.061614 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:04.061446    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerDied","Data":"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"}
Jul 31 15:05:04.063707 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:04.063681    2575 generic.go:356] "Generic (PLEG): container finished" podID="93b19444-3315-44f0-8112-c1f1513f97ed" containerID="c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5" exitCode=2
Jul 31 15:05:04.063848 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:04.063720    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerDied","Data":"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"}
Jul 31 15:05:05.474736 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:05.474689    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 31 15:05:05.479981 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:05.479950    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:05:06.021531 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:06.021488    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:06.480666 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:06.480626    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 31 15:05:06.486158 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:06.486120    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:05:07.727098 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.727060    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:05:07.779729 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.779697    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config\") pod \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") "
Jul 31 15:05:07.779914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.779752    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert\") pod \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") "
Jul 31 15:05:07.779914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.779775    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6p7c\" (UniqueName: \"kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c\") pod \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") "
Jul 31 15:05:07.779914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.779811    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls\") pod \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") "
Jul 31 15:05:07.779914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.779831    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location\") pod \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\" (UID: \"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f\") "
Jul 31 15:05:07.780154 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.780112    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" (UID: "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:07.780256 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.780229    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" (UID: "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:07.780256 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.780235    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" (UID: "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:07.782127 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.782105    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c" (OuterVolumeSpecName: "kube-api-access-z6p7c") pod "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" (UID: "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f"). InnerVolumeSpecName "kube-api-access-z6p7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:05:07.782127 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.782104    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" (UID: "c72ab21f-8fd4-4b76-ad0b-94a251a5d62f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:05:07.880865 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.880823    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6p7c\" (UniqueName: \"kubernetes.io/projected/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-api-access-z6p7c\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:07.880865 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.880858    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:07.880865 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.880870    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:07.880865 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.880880    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:07.881149 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:07.880889    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:08.086380 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.086287    2575 generic.go:356] "Generic (PLEG): container finished" podID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerID="2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8" exitCode=0
Jul 31 15:05:08.086380 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.086349    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerDied","Data":"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"}
Jul 31 15:05:08.086380 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.086352    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"
Jul 31 15:05:08.086650 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.086399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n" event={"ID":"c72ab21f-8fd4-4b76-ad0b-94a251a5d62f","Type":"ContainerDied","Data":"ae0526a127f45273a0625bc90c0e0613b0de478e6a86e81c0a73de943fd5ccc8"}
Jul 31 15:05:08.086650 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.086427    2575 scope.go:117] "RemoveContainer" containerID="f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"
Jul 31 15:05:08.096372 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.096348    2575 scope.go:117] "RemoveContainer" containerID="2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"
Jul 31 15:05:08.106008 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.105983    2575 scope.go:117] "RemoveContainer" containerID="cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"
Jul 31 15:05:08.110677 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.110639    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:05:08.115566 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.115533    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e936d-predictor-5447cb98bf-9449n"]
Jul 31 15:05:08.116309 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.116293    2575 scope.go:117] "RemoveContainer" containerID="f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"
Jul 31 15:05:08.116676 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:08.116650    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5\": container with ID starting with f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5 not found: ID does not exist" containerID="f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"
Jul 31 15:05:08.116781 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.116686    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5"} err="failed to get container status \"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5\": rpc error: code = NotFound desc = could not find container \"f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5\": container with ID starting with f61b7f7999da86523a83fb319acba535781b744b29fd7af6758fb584013a8ad5 not found: ID does not exist"
Jul 31 15:05:08.116781 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.116708    2575 scope.go:117] "RemoveContainer" containerID="2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"
Jul 31 15:05:08.116993 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:08.116975    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8\": container with ID starting with 2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8 not found: ID does not exist" containerID="2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"
Jul 31 15:05:08.117040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.117001    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8"} err="failed to get container status \"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8\": rpc error: code = NotFound desc = could not find container \"2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8\": container with ID starting with 2bbbd19fdb0ca790c31a59128df91527b60488bd4e26ff850c7ada4715c305a8 not found: ID does not exist"
Jul 31 15:05:08.117040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.117017    2575 scope.go:117] "RemoveContainer" containerID="cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"
Jul 31 15:05:08.117264 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:08.117247    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a\": container with ID starting with cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a not found: ID does not exist" containerID="cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"
Jul 31 15:05:08.117311 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.117270    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a"} err="failed to get container status \"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a\": rpc error: code = NotFound desc = could not find container \"cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a\": container with ID starting with cbc552e7f407e7054ac5a17e1cf45fb3fd6509759c3f505e16e42bdec020fb3a not found: ID does not exist"
Jul 31 15:05:08.898452 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.898422    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:05:08.991106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991014    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location\") pod \"93b19444-3315-44f0-8112-c1f1513f97ed\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") "
Jul 31 15:05:08.991106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991097    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert\") pod \"93b19444-3315-44f0-8112-c1f1513f97ed\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") "
Jul 31 15:05:08.991358 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991120    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config\") pod \"93b19444-3315-44f0-8112-c1f1513f97ed\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") "
Jul 31 15:05:08.991358 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991151    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls\") pod \"93b19444-3315-44f0-8112-c1f1513f97ed\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") "
Jul 31 15:05:08.991358 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991173    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sttp8\" (UniqueName: \"kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8\") pod \"93b19444-3315-44f0-8112-c1f1513f97ed\" (UID: \"93b19444-3315-44f0-8112-c1f1513f97ed\") "
Jul 31 15:05:08.991514 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991393    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93b19444-3315-44f0-8112-c1f1513f97ed" (UID: "93b19444-3315-44f0-8112-c1f1513f97ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:08.991569 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93b19444-3315-44f0-8112-c1f1513f97ed" (UID: "93b19444-3315-44f0-8112-c1f1513f97ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:08.991569 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.991527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93b19444-3315-44f0-8112-c1f1513f97ed" (UID: "93b19444-3315-44f0-8112-c1f1513f97ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:08.993479 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.993457    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93b19444-3315-44f0-8112-c1f1513f97ed" (UID: "93b19444-3315-44f0-8112-c1f1513f97ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:05:08.993479 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:08.993466    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8" (OuterVolumeSpecName: "kube-api-access-sttp8") pod "93b19444-3315-44f0-8112-c1f1513f97ed" (UID: "93b19444-3315-44f0-8112-c1f1513f97ed"). InnerVolumeSpecName "kube-api-access-sttp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:05:09.091988 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.091955    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93b19444-3315-44f0-8112-c1f1513f97ed-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:09.091988 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.091982    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:09.091988 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.091992    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93b19444-3315-44f0-8112-c1f1513f97ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:09.092245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.092001    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93b19444-3315-44f0-8112-c1f1513f97ed-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:09.092245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.092010    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sttp8\" (UniqueName: \"kubernetes.io/projected/93b19444-3315-44f0-8112-c1f1513f97ed-kube-api-access-sttp8\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:09.093492 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.093461    2575 generic.go:356] "Generic (PLEG): container finished" podID="93b19444-3315-44f0-8112-c1f1513f97ed" containerID="ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f" exitCode=0
Jul 31 15:05:09.093626 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.093522    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"
Jul 31 15:05:09.093626 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.093566    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerDied","Data":"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"}
Jul 31 15:05:09.093720 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.093661    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml" event={"ID":"93b19444-3315-44f0-8112-c1f1513f97ed","Type":"ContainerDied","Data":"a60a095804315808391ed2841a0eaac71f91cabb878bcccef311cca47f446bbe"}
Jul 31 15:05:09.093720 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.093700    2575 scope.go:117] "RemoveContainer" containerID="c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"
Jul 31 15:05:09.102872 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.102849    2575 scope.go:117] "RemoveContainer" containerID="ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"
Jul 31 15:05:09.111859 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.111836    2575 scope.go:117] "RemoveContainer" containerID="65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"
Jul 31 15:05:09.117344 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.117314    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:05:09.121556 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.121497    2575 scope.go:117] "RemoveContainer" containerID="c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"
Jul 31 15:05:09.121884 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:09.121859    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5\": container with ID starting with c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5 not found: ID does not exist" containerID="c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"
Jul 31 15:05:09.121999 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.121890    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5"} err="failed to get container status \"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5\": rpc error: code = NotFound desc = could not find container \"c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5\": container with ID starting with c4ab83aa5638adb5926e42dec2bd503cfe6b26e585099fbdd0b41a5d41e74fa5 not found: ID does not exist"
Jul 31 15:05:09.121999 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.121909    2575 scope.go:117] "RemoveContainer" containerID="ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"
Jul 31 15:05:09.122199 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.122182    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e936d-predictor-5cc88b69ff-42nml"]
Jul 31 15:05:09.122246 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:09.122204    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f\": container with ID starting with ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f not found: ID does not exist" containerID="ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"
Jul 31 15:05:09.122246 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.122228    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f"} err="failed to get container status \"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f\": rpc error: code = NotFound desc = could not find container \"ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f\": container with ID starting with ad2b5e1b72f89e70b11169bb524055b9c3899dacec46e9b81fb9204c26982a5f not found: ID does not exist"
Jul 31 15:05:09.122246 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.122243    2575 scope.go:117] "RemoveContainer" containerID="65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"
Jul 31 15:05:09.122531 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:09.122510    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b\": container with ID starting with 65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b not found: ID does not exist" containerID="65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"
Jul 31 15:05:09.122591 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.122539    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b"} err="failed to get container status \"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b\": rpc error: code = NotFound desc = could not find container \"65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b\": container with ID starting with 65e61dd5abc5cf0bc47e43154f05bc700a2a4fbc8c3759a5626ddb0c31fe895b not found: ID does not exist"
Jul 31 15:05:09.206387 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.206350    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" path="/var/lib/kubelet/pods/93b19444-3315-44f0-8112-c1f1513f97ed/volumes"
Jul 31 15:05:09.206875 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:09.206858    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" path="/var/lib/kubelet/pods/c72ab21f-8fd4-4b76-ad0b-94a251a5d62f/volumes"
Jul 31 15:05:11.020618 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:11.020553    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:16.020662 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:16.020589    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:16.021058 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:16.020743    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:05:21.020902 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:21.020857    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:23.511752 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.511714    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:23.545789 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.545750    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:05:23.546078 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546067    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546080    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="storage-initializer"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546098    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="storage-initializer"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546110    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="storage-initializer"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546116    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="storage-initializer"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546129    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546134 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546136    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546145    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546150    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546156    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546162    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546208    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546218    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kserve-container"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546224    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b19444-3315-44f0-8112-c1f1513f97ed" containerName="kube-rbac-proxy"
Jul 31 15:05:23.546445 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.546231    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72ab21f-8fd4-4b76-ad0b-94a251a5d62f" containerName="kserve-container"
Jul 31 15:05:23.549385 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.549360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.552133 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.552096    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-126e1-kube-rbac-proxy-sar-config\""
Jul 31 15:05:23.552362 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.552103    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:05:23.552362 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.552201    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-126e1-predictor-serving-cert\""
Jul 31 15:05:23.552362 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.552214    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:05:23.552625 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.552593    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:05:23.561109 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.561071    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:05:23.611168 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.611126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.611360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.611177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmcz\" (UniqueName: \"kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.611360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.611205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.611360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.611251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.611360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.611294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712142 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712085    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712142 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmcz\" (UniqueName: \"kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712437 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:23.712264    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-serving-cert: secret "isvc-logger-raw-126e1-predictor-serving-cert" not found
Jul 31 15:05:23.712437 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:23.712331    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls podName:1dc92568-1ba5-430b-a859-55fc674871da nodeName:}" failed. No retries permitted until 2026-07-31 15:05:24.212312233 +0000 UTC m=+1047.546432134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls") pod "isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" (UID: "1dc92568-1ba5-430b-a859-55fc674871da") : secret "isvc-logger-raw-126e1-predictor-serving-cert" not found
Jul 31 15:05:23.712437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712271    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.712714 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.713013 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.712988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.713095 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.713079    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.721386 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.721358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmcz\" (UniqueName: \"kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:23.960105 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:23.960055    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:24.216706 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:24.216577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:24.219210 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:24.219179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") pod \"isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") " pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:24.461463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:24.461418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:24.611698 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:24.611660    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:05:24.613037 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:05:24.613009    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc92568_1ba5_430b_a859_55fc674871da.slice/crio-07cafb82d395ec533080a0e8418b5eb9d7d743c715642d4cabe3af2d42c3bf15 WatchSource:0}: Error finding container 07cafb82d395ec533080a0e8418b5eb9d7d743c715642d4cabe3af2d42c3bf15: Status 404 returned error can't find the container with id 07cafb82d395ec533080a0e8418b5eb9d7d743c715642d4cabe3af2d42c3bf15
Jul 31 15:05:25.171120 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:25.171072    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerStarted","Data":"02acd25e1af93b1b76407ebac2b721a6aa9f81a3ed820d0cea186e412547943d"}
Jul 31 15:05:25.171339 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:25.171167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerStarted","Data":"07cafb82d395ec533080a0e8418b5eb9d7d743c715642d4cabe3af2d42c3bf15"}
Jul 31 15:05:26.020774 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:26.020726    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:29.192130 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:29.192094    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dc92568-1ba5-430b-a859-55fc674871da" containerID="02acd25e1af93b1b76407ebac2b721a6aa9f81a3ed820d0cea186e412547943d" exitCode=0
Jul 31 15:05:29.192518 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:29.192180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerDied","Data":"02acd25e1af93b1b76407ebac2b721a6aa9f81a3ed820d0cea186e412547943d"}
Jul 31 15:05:30.198955 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.198915    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerStarted","Data":"cba29875474cd344756e25f0e64e5a1f52eae7c8a098f53506d59fb4ad5d00bf"}
Jul 31 15:05:30.199352 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.198995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerStarted","Data":"86811824a8a1f2e5c9607abadb3fc7c787e8343570ccb7b0405806c997f9cae1"}
Jul 31 15:05:30.199352 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.199053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerStarted","Data":"16abc36233bec85a3211c8fa84277d5410eed327e9da933979abb5e2b8bdca07"}
Jul 31 15:05:30.199352 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.199269    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:30.199352 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.199329    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:30.200943 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.200915    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:30.223146 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:30.223085    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podStartSLOduration=3.147474413 podStartE2EDuration="7.22306802s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.075593607s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:30.222405514 +0000 UTC m=+1053.556525436" watchObservedRunningTime="2026-07-31 15:05:30.22306802 +0000 UTC m=+1053.557187941"
Jul 31 15:05:31.020574 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:31.020524    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:31.203351 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:31.203317    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:31.203845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:31.203439    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:31.204686 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:31.204653    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:32.213128 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:32.213068    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:32.213641 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:32.213563    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:33.485892 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.485858    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:05:33.597037 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.596926    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle\") pod \"4320da7a-a254-4d03-87a3-f9ff9115fc55\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") "
Jul 31 15:05:33.597037 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.596979    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") pod \"4320da7a-a254-4d03-87a3-f9ff9115fc55\" (UID: \"4320da7a-a254-4d03-87a3-f9ff9115fc55\") "
Jul 31 15:05:33.597309 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.597284    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4320da7a-a254-4d03-87a3-f9ff9115fc55" (UID: "4320da7a-a254-4d03-87a3-f9ff9115fc55"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:33.599244 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.599209    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4320da7a-a254-4d03-87a3-f9ff9115fc55" (UID: "4320da7a-a254-4d03-87a3-f9ff9115fc55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:05:33.698467 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.698424    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4320da7a-a254-4d03-87a3-f9ff9115fc55-openshift-service-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:33.698467 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:33.698456    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4320da7a-a254-4d03-87a3-f9ff9115fc55-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:05:34.222929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.222889    2575 generic.go:356] "Generic (PLEG): container finished" podID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerID="b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08" exitCode=0
Jul 31 15:05:34.223090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.222941    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"
Jul 31 15:05:34.223090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.222980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" event={"ID":"4320da7a-a254-4d03-87a3-f9ff9115fc55","Type":"ContainerDied","Data":"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"}
Jul 31 15:05:34.223090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.223038    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj" event={"ID":"4320da7a-a254-4d03-87a3-f9ff9115fc55","Type":"ContainerDied","Data":"fb279b140a59d078a3fab57ee71ad13e050c948e558fef4273490f723a365414"}
Jul 31 15:05:34.223090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.223062    2575 scope.go:117] "RemoveContainer" containerID="b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"
Jul 31 15:05:34.232886 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.232862    2575 scope.go:117] "RemoveContainer" containerID="b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"
Jul 31 15:05:34.233216 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:05:34.233194    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08\": container with ID starting with b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08 not found: ID does not exist" containerID="b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"
Jul 31 15:05:34.233303 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.233226    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08"} err="failed to get container status \"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08\": rpc error: code = NotFound desc = could not find container \"b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08\": container with ID starting with b67ae514ac5be75504b2485755d88f7e22df49eacd5c16253b984d5636abef08 not found: ID does not exist"
Jul 31 15:05:34.246003 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.245960    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:05:34.251839 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.251806    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e936d-84649d8867-c2cdj"]
Jul 31 15:05:34.272876 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:34.272847    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:05:35.202081 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:35.202041    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" path="/var/lib/kubelet/pods/4320da7a-a254-4d03-87a3-f9ff9115fc55/volumes"
Jul 31 15:05:37.218156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:37.218126    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:05:37.218831 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:37.218797    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:37.219062 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:37.219033    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:47.219624 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:47.219543    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:47.220124 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:47.220073    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:05:57.218810 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:57.218770    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:05:57.219295 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:05:57.219264    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:07.218837 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:07.218802    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:06:07.219316 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:07.219049    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:17.219089 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:17.219028    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:06:17.219531 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:17.219509    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:18.377827 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.377783    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:18.417315 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.417273    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:18.417679 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.417664    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d"
Jul 31 15:06:18.417751 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.417681    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d"
Jul 31 15:06:18.417789 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.417760    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4320da7a-a254-4d03-87a3-f9ff9115fc55" containerName="model-chainer-raw-hpa-e936d"
Jul 31 15:06:18.421154 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.421129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.424724 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.424693    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-74ffcf-dockercfg-dgm9r\""
Jul 31 15:06:18.424905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.424693    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-74ffcf-predictor-serving-cert\""
Jul 31 15:06:18.424966 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.424944    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-74ffcf-kube-rbac-proxy-sar-config\""
Jul 31 15:06:18.425798 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.425772    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-74ffcf\""
Jul 31 15:06:18.433481 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.433451    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:18.479825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.479779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.479825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.479825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8nt\" (UniqueName: \"kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.480060 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.479960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.480060 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.480001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.480060 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.480030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581170 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581115    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581170 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8nt\" (UniqueName: \"kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581463 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:18.581394    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-serving-cert: secret "isvc-secondary-74ffcf-predictor-serving-cert" not found
Jul 31 15:06:18.581745 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:18.581486    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls podName:d5d6acc2-48d4-4555-900a-3f211de589bd nodeName:}" failed. No retries permitted until 2026-07-31 15:06:19.081460912 +0000 UTC m=+1102.415580826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls") pod "isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd") : secret "isvc-secondary-74ffcf-predictor-serving-cert" not found
Jul 31 15:06:18.581745 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581594    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.581880 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.581868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.582035 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.582012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.590741 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.590702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8nt\" (UniqueName: \"kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:18.678022 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:18.677913    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:19.086495 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:19.086374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:19.089053 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:19.089026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") pod \"isvc-secondary-74ffcf-predictor-6b558798b9-wgw69\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") " pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:19.333576 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:19.333530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:19.478636 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:19.478428    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:19.481143 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:06:19.481109    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d6acc2_48d4_4555_900a_3f211de589bd.slice/crio-0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162 WatchSource:0}: Error finding container 0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162: Status 404 returned error can't find the container with id 0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162
Jul 31 15:06:20.434234 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:20.434192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerStarted","Data":"83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"}
Jul 31 15:06:20.434480 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:20.434260    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerStarted","Data":"0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162"}
Jul 31 15:06:21.544119 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:21.544061    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:24.454047 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:24.454016    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/0.log"
Jul 31 15:06:24.454439 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:24.454056    2575 generic.go:356] "Generic (PLEG): container finished" podID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerID="83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571" exitCode=1
Jul 31 15:06:24.454439 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:24.454111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerDied","Data":"83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"}
Jul 31 15:06:25.460407 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:25.460371    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/0.log"
Jul 31 15:06:25.460877 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:25.460502    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerStarted","Data":"08dce5fb37974c0e4cc676f9faac492fa41e643b2ba8536a5893475287225c36"}
Jul 31 15:06:27.218998 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:27.218956    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:06:27.219449 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:27.219366    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:28.475856 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.475822    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/1.log"
Jul 31 15:06:28.476249 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.476232    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/0.log"
Jul 31 15:06:28.476316 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.476266    2575 generic.go:356] "Generic (PLEG): container finished" podID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerID="08dce5fb37974c0e4cc676f9faac492fa41e643b2ba8536a5893475287225c36" exitCode=1
Jul 31 15:06:28.476368 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.476344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerDied","Data":"08dce5fb37974c0e4cc676f9faac492fa41e643b2ba8536a5893475287225c36"}
Jul 31 15:06:28.476424 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.476414    2575 scope.go:117] "RemoveContainer" containerID="83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"
Jul 31 15:06:28.476799 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.476784    2575 scope.go:117] "RemoveContainer" containerID="83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"
Jul 31 15:06:28.488209 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:28.488184    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_kserve-ci-e2e-test(d5d6acc2-48d4-4555-900a-3f211de589bd)\"" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd"
Jul 31 15:06:28.488309 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:28.488261    2575 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_kserve-ci-e2e-test_d5d6acc2-48d4-4555-900a-3f211de589bd_0 in pod sandbox 0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162 from index: no such id: '83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571'" containerID="83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"
Jul 31 15:06:28.488366 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:28.488311    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_kserve-ci-e2e-test_d5d6acc2-48d4-4555-900a-3f211de589bd_0 in pod sandbox 0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162 from index: no such id: '83bb63f05c737ff3e7d2506f7ade826b4494507217586e19217b28057a412571'"
Jul 31 15:06:29.482551 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:29.482519    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/1.log"
Jul 31 15:06:32.344471 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:32.344413    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:32.405206 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:32.405107    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:36.446333 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.446291    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:36.468787 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.468749    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:36.535293 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.535253    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:36.586454 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.586431    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/1.log"
Jul 31 15:06:36.590176 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.590151    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:36.737995 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.737891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config\") pod \"d5d6acc2-48d4-4555-900a-3f211de589bd\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") "
Jul 31 15:06:36.737995 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.737949    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert\") pod \"d5d6acc2-48d4-4555-900a-3f211de589bd\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") "
Jul 31 15:06:36.737995 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.737997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") pod \"d5d6acc2-48d4-4555-900a-3f211de589bd\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") "
Jul 31 15:06:36.738305 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.738087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d8nt\" (UniqueName: \"kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt\") pod \"d5d6acc2-48d4-4555-900a-3f211de589bd\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") "
Jul 31 15:06:36.738305 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.738119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location\") pod \"d5d6acc2-48d4-4555-900a-3f211de589bd\" (UID: \"d5d6acc2-48d4-4555-900a-3f211de589bd\") "
Jul 31 15:06:36.738305 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.738289    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5d6acc2-48d4-4555-900a-3f211de589bd" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:36.738460 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.738420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5d6acc2-48d4-4555-900a-3f211de589bd" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:36.738460 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.738428    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5d6acc2-48d4-4555-900a-3f211de589bd" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:36.740506 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.740462    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt" (OuterVolumeSpecName: "kube-api-access-6d8nt") pod "d5d6acc2-48d4-4555-900a-3f211de589bd" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd"). InnerVolumeSpecName "kube-api-access-6d8nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:36.740506 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.740471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5d6acc2-48d4-4555-900a-3f211de589bd" (UID: "d5d6acc2-48d4-4555-900a-3f211de589bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:36.838829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.838795    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:36.838829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.838825    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5d6acc2-48d4-4555-900a-3f211de589bd-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:36.838829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.838836    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8nt\" (UniqueName: \"kubernetes.io/projected/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-api-access-6d8nt\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:36.838829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.838845    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5d6acc2-48d4-4555-900a-3f211de589bd-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:36.839121 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:36.838856    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5d6acc2-48d4-4555-900a-3f211de589bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:37.219522 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.219472    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:06:37.220009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.219983    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:37.522436 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.522354    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-74ffcf-predictor-6b558798b9-wgw69_d5d6acc2-48d4-4555-900a-3f211de589bd/storage-initializer/1.log"
Jul 31 15:06:37.522845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.522437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69" event={"ID":"d5d6acc2-48d4-4555-900a-3f211de589bd","Type":"ContainerDied","Data":"0b2972db846d08c9ff3e5a16a21382bbf953643053a0d0e693b1b71751f2a162"}
Jul 31 15:06:37.522845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.522477    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"
Jul 31 15:06:37.522845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.522521    2575 scope.go:117] "RemoveContainer" containerID="08dce5fb37974c0e4cc676f9faac492fa41e643b2ba8536a5893475287225c36"
Jul 31 15:06:37.561087 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.561047    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:37.563544 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:37.563511    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-74ffcf-predictor-6b558798b9-wgw69"]
Jul 31 15:06:39.202093 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:39.202056    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" path="/var/lib/kubelet/pods/d5d6acc2-48d4-4555-900a-3f211de589bd/volumes"
Jul 31 15:06:41.628222 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.628187    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:41.663066 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663026    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:41.663364 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663351    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.663409 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663366    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.663463 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663454    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.663499 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663465    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.663532 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663519    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.663532 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.663525    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d6acc2-48d4-4555-900a-3f211de589bd" containerName="storage-initializer"
Jul 31 15:06:41.668003 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.667979    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.670694 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.670664    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9fe2cb-predictor-serving-cert\""
Jul 31 15:06:41.670694 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.670668    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9fe2cb\""
Jul 31 15:06:41.670893 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.670811    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9fe2cb-kube-rbac-proxy-sar-config\""
Jul 31 15:06:41.670893 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.670823    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9fe2cb-dockercfg-49mqs\""
Jul 31 15:06:41.677895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.677866    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:41.777011 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.776944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.777011 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.777007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqldn\" (UniqueName: \"kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.777279 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.777077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.777279 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.777106    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.777279 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.777128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.878511 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.878394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.878511 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.878446    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.878511 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.878469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.878901 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.878688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.878901 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.878736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqldn\" (UniqueName: \"kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.879074 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.879051    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.879250 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.879230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.879324 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.879266    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.881190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.881163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.893668 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.893624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqldn\" (UniqueName: \"kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn\") pod \"isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:41.938939 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.938895    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:41.979572 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:41.979529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:42.132769 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:42.132528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:42.135710 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:06:42.135671    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac6655d_a248_4df1_b722_c5794263917e.slice/crio-81d81789f02c9fc152ceb6046134864c77d65bf1406253eaee3df4fd5e55acf9 WatchSource:0}: Error finding container 81d81789f02c9fc152ceb6046134864c77d65bf1406253eaee3df4fd5e55acf9: Status 404 returned error can't find the container with id 81d81789f02c9fc152ceb6046134864c77d65bf1406253eaee3df4fd5e55acf9
Jul 31 15:06:42.549539 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:42.549449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerStarted","Data":"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"}
Jul 31 15:06:42.549539 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:42.549507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerStarted","Data":"81d81789f02c9fc152ceb6046134864c77d65bf1406253eaee3df4fd5e55acf9"}
Jul 31 15:06:46.196877 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:46.196817    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:06:46.197289 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:46.197224    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:06:49.584104 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:49.584077    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/0.log"
Jul 31 15:06:49.584503 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:49.584117    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ac6655d-a248-4df1-b722-c5794263917e" containerID="bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2" exitCode=1
Jul 31 15:06:49.584503 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:49.584180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerDied","Data":"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"}
Jul 31 15:06:50.591334 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:50.591305    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/0.log"
Jul 31 15:06:50.591807 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:50.591401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerStarted","Data":"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"}
Jul 31 15:06:51.661090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:51.661053    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:51.661634 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:51.661274    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer" containerID="cri-o://01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a" gracePeriod=30
Jul 31 15:06:51.692280 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:51.692243    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:53.320060 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.320036    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/1.log"
Jul 31 15:06:53.320468 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.320454    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/0.log"
Jul 31 15:06:53.322888 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.322863    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:53.482180 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482079    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert\") pod \"5ac6655d-a248-4df1-b722-c5794263917e\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") "
Jul 31 15:06:53.482180 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482133    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location\") pod \"5ac6655d-a248-4df1-b722-c5794263917e\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") "
Jul 31 15:06:53.482180 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482161    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls\") pod \"5ac6655d-a248-4df1-b722-c5794263917e\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") "
Jul 31 15:06:53.482429 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482260    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqldn\" (UniqueName: \"kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn\") pod \"5ac6655d-a248-4df1-b722-c5794263917e\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") "
Jul 31 15:06:53.482429 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482291    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config\") pod \"5ac6655d-a248-4df1-b722-c5794263917e\" (UID: \"5ac6655d-a248-4df1-b722-c5794263917e\") "
Jul 31 15:06:53.482535 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482457    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ac6655d-a248-4df1-b722-c5794263917e" (UID: "5ac6655d-a248-4df1-b722-c5794263917e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:53.482535 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482523    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ac6655d-a248-4df1-b722-c5794263917e" (UID: "5ac6655d-a248-4df1-b722-c5794263917e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:53.482687 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482630    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6655d-a248-4df1-b722-c5794263917e-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:53.482784 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.482761    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ac6655d-a248-4df1-b722-c5794263917e" (UID: "5ac6655d-a248-4df1-b722-c5794263917e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:53.484636 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.484589    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ac6655d-a248-4df1-b722-c5794263917e" (UID: "5ac6655d-a248-4df1-b722-c5794263917e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:53.484750 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.484693    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn" (OuterVolumeSpecName: "kube-api-access-sqldn") pod "5ac6655d-a248-4df1-b722-c5794263917e" (UID: "5ac6655d-a248-4df1-b722-c5794263917e"). InnerVolumeSpecName "kube-api-access-sqldn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:53.583555 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.583512    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac6655d-a248-4df1-b722-c5794263917e-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:53.583555 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.583549    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqldn\" (UniqueName: \"kubernetes.io/projected/5ac6655d-a248-4df1-b722-c5794263917e-kube-api-access-sqldn\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:53.583555 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.583559    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:53.583555 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.583569    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6655d-a248-4df1-b722-c5794263917e-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:06:53.607863 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.607838    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/1.log"
Jul 31 15:06:53.608268 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj_5ac6655d-a248-4df1-b722-c5794263917e/storage-initializer/0.log"
Jul 31 15:06:53.608313 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608290    2575 generic.go:356] "Generic (PLEG): container finished" podID="5ac6655d-a248-4df1-b722-c5794263917e" containerID="01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a" exitCode=1
Jul 31 15:06:53.608354 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608321    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerDied","Data":"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"}
Jul 31 15:06:53.608385 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608353    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"
Jul 31 15:06:53.608418 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608387    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj" event={"ID":"5ac6655d-a248-4df1-b722-c5794263917e","Type":"ContainerDied","Data":"81d81789f02c9fc152ceb6046134864c77d65bf1406253eaee3df4fd5e55acf9"}
Jul 31 15:06:53.608418 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.608415    2575 scope.go:117] "RemoveContainer" containerID="01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"
Jul 31 15:06:53.617754 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.617731    2575 scope.go:117] "RemoveContainer" containerID="bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"
Jul 31 15:06:53.626706 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.626680    2575 scope.go:117] "RemoveContainer" containerID="01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"
Jul 31 15:06:53.627052 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:53.627029    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a\": container with ID starting with 01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a not found: ID does not exist" containerID="01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"
Jul 31 15:06:53.627116 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.627064    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a"} err="failed to get container status \"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a\": rpc error: code = NotFound desc = could not find container \"01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a\": container with ID starting with 01df7d5edeb69b093d5c03c80febf05736de1405c5aacb2113b2784799ff2f4a not found: ID does not exist"
Jul 31 15:06:53.627116 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.627085    2575 scope.go:117] "RemoveContainer" containerID="bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"
Jul 31 15:06:53.627368 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:06:53.627346    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2\": container with ID starting with bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2 not found: ID does not exist" containerID="bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"
Jul 31 15:06:53.627476 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.627371    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2"} err="failed to get container status \"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2\": rpc error: code = NotFound desc = could not find container \"bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2\": container with ID starting with bb07a1f416237147d4c900f8e102d72d70db0a3e3a020012d39816f8211adad2 not found: ID does not exist"
Jul 31 15:06:53.649365 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.649324    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:53.652054 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:53.652020    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9fe2cb-predictor-6bd4c75665-bh2rj"]
Jul 31 15:06:55.201873 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:55.201840    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac6655d-a248-4df1-b722-c5794263917e" path="/var/lib/kubelet/pods/5ac6655d-a248-4df1-b722-c5794263917e/volumes"
Jul 31 15:06:56.197627 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:56.197566    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:06:56.197841 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:56.197745    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:06:56.948399 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:56.948362    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:57.000930 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.000897    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:06:57.001241 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001229    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.001284 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001243    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.001284 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001261    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.001284 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001267    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.001384 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001322    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.001384 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.001333    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac6655d-a248-4df1-b722-c5794263917e" containerName="storage-initializer"
Jul 31 15:06:57.006044 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.006013    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.008827 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.008791    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2d318-predictor-serving-cert\""
Jul 31 15:06:57.009240 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.009225    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2d318-kube-rbac-proxy-sar-config\""
Jul 31 15:06:57.018439 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.018408    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:06:57.112953 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.112911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.112953 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.112960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.113218 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.112984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.113218 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.113047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.113218 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.113109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zwc\" (UniqueName: \"kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.183500 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.183406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:06:57.213659 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.213624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.213843 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.213706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zwc\" (UniqueName: \"kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.213843 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.213789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.213927 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.213843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.213927 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.213879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.214292 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.214270    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.214561 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.214542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.216430 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.216406    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2d318-kube-rbac-proxy-sar-config\""
Jul 31 15:06:57.216568 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.216499    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-2d318-predictor-serving-cert\""
Jul 31 15:06:57.222736 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.222707    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zwc\" (UniqueName: \"kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.225620 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.225474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.227584 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.227547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls\") pod \"raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") " pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.317189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.317157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:06:57.465660 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.465632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:06:57.468334 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:06:57.468308    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ae3d97_b27f_4234_a6c5_ed952ea65499.slice/crio-d92045276fe70bd3492150716909ba9e70b0a06e4ae46315fa1e1f3d5b2554c7 WatchSource:0}: Error finding container d92045276fe70bd3492150716909ba9e70b0a06e4ae46315fa1e1f3d5b2554c7: Status 404 returned error can't find the container with id d92045276fe70bd3492150716909ba9e70b0a06e4ae46315fa1e1f3d5b2554c7
Jul 31 15:06:57.629497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.629463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerStarted","Data":"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"}
Jul 31 15:06:57.629679 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:06:57.629524    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerStarted","Data":"d92045276fe70bd3492150716909ba9e70b0a06e4ae46315fa1e1f3d5b2554c7"}
Jul 31 15:07:01.649491 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:01.649452    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerID="827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d" exitCode=0
Jul 31 15:07:01.649918 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:01.649523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerDied","Data":"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"}
Jul 31 15:07:02.658664 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:02.658626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerStarted","Data":"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"}
Jul 31 15:07:02.659059 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:02.658695    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerStarted","Data":"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"}
Jul 31 15:07:02.659059 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:02.658885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:07:02.684557 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:02.684499    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podStartSLOduration=2.901021865 podStartE2EDuration="6.684480746s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783458881s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:07:02.682416708 +0000 UTC m=+1146.016536656" watchObservedRunningTime="2026-07-31 15:07:02.684480746 +0000 UTC m=+1146.018600720"
Jul 31 15:07:03.663644 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:03.663587    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:07:03.665200 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:03.665174    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:04.667901 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:04.667855    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:08.810327 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.810283    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:07:08.810805 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.810624    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" containerID="cri-o://16abc36233bec85a3211c8fa84277d5410eed327e9da933979abb5e2b8bdca07" gracePeriod=30
Jul 31 15:07:08.810805 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.810657    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" containerID="cri-o://cba29875474cd344756e25f0e64e5a1f52eae7c8a098f53506d59fb4ad5d00bf" gracePeriod=30
Jul 31 15:07:08.810805 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.810675    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" containerID="cri-o://86811824a8a1f2e5c9607abadb3fc7c787e8343570ccb7b0405806c997f9cae1" gracePeriod=30
Jul 31 15:07:08.822287 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.822227    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:07:08.905157 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:08.905122    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:07:09.598236 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.598199    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:07:09.674028 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.673998    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:07:09.674772 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.674746    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:09.694635 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.694582    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dc92568-1ba5-430b-a859-55fc674871da" containerID="86811824a8a1f2e5c9607abadb3fc7c787e8343570ccb7b0405806c997f9cae1" exitCode=2
Jul 31 15:07:09.694815 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.694664    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerDied","Data":"86811824a8a1f2e5c9607abadb3fc7c787e8343570ccb7b0405806c997f9cae1"}
Jul 31 15:07:09.699002 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:09.698973    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:07:12.214075 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:12.214030    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:14.720162 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:14.720125    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dc92568-1ba5-430b-a859-55fc674871da" containerID="16abc36233bec85a3211c8fa84277d5410eed327e9da933979abb5e2b8bdca07" exitCode=0
Jul 31 15:07:14.720579 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:14.720180    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerDied","Data":"16abc36233bec85a3211c8fa84277d5410eed327e9da933979abb5e2b8bdca07"}
Jul 31 15:07:16.197095 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:16.197041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:07:16.197525 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:16.197385    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:07:17.213749 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:17.213704    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:19.675643 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:19.675523    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:22.213714 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:22.213662    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:22.214105 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:22.213802    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:07:26.197092 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:26.197036    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:07:26.197490 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:26.197388    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:07:27.213623 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:27.213551    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:29.675197 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:29.675155    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:32.214071 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:32.214023    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:36.196802 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:36.196753    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:07:36.197226 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:36.196928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:07:36.197226 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:36.197196    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:07:36.197330 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:36.197308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:07:37.213503 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:37.213459    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:07:38.842496 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:38.842465    2575 generic.go:356] "Generic (PLEG): container finished" podID="1dc92568-1ba5-430b-a859-55fc674871da" containerID="cba29875474cd344756e25f0e64e5a1f52eae7c8a098f53506d59fb4ad5d00bf" exitCode=0
Jul 31 15:07:38.842901 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:38.842536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerDied","Data":"cba29875474cd344756e25f0e64e5a1f52eae7c8a098f53506d59fb4ad5d00bf"}
Jul 31 15:07:38.978285 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:38.978243    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:07:39.072348 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072237    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csmcz\" (UniqueName: \"kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz\") pod \"1dc92568-1ba5-430b-a859-55fc674871da\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") "
Jul 31 15:07:39.072348 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072294    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config\") pod \"1dc92568-1ba5-430b-a859-55fc674871da\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") "
Jul 31 15:07:39.072348 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072330    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") pod \"1dc92568-1ba5-430b-a859-55fc674871da\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") "
Jul 31 15:07:39.072348 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072346    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert\") pod \"1dc92568-1ba5-430b-a859-55fc674871da\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") "
Jul 31 15:07:39.072765 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072385    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location\") pod \"1dc92568-1ba5-430b-a859-55fc674871da\" (UID: \"1dc92568-1ba5-430b-a859-55fc674871da\") "
Jul 31 15:07:39.072828 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072800    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1dc92568-1ba5-430b-a859-55fc674871da" (UID: "1dc92568-1ba5-430b-a859-55fc674871da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:07:39.072828 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072810    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1dc92568-1ba5-430b-a859-55fc674871da" (UID: "1dc92568-1ba5-430b-a859-55fc674871da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:07:39.072905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.072816    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dc92568-1ba5-430b-a859-55fc674871da" (UID: "1dc92568-1ba5-430b-a859-55fc674871da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:39.074727 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.074691    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dc92568-1ba5-430b-a859-55fc674871da" (UID: "1dc92568-1ba5-430b-a859-55fc674871da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:07:39.074727 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.074709    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz" (OuterVolumeSpecName: "kube-api-access-csmcz") pod "1dc92568-1ba5-430b-a859-55fc674871da" (UID: "1dc92568-1ba5-430b-a859-55fc674871da"). InnerVolumeSpecName "kube-api-access-csmcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:07:39.173229 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.173192    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csmcz\" (UniqueName: \"kubernetes.io/projected/1dc92568-1ba5-430b-a859-55fc674871da-kube-api-access-csmcz\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:07:39.173229 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.173226    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:07:39.173229 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.173236    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dc92568-1ba5-430b-a859-55fc674871da-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:07:39.173468 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.173245    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1dc92568-1ba5-430b-a859-55fc674871da-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:07:39.173468 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.173253    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dc92568-1ba5-430b-a859-55fc674871da-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:07:39.675709 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.675657    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:39.849894 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.849848    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv" event={"ID":"1dc92568-1ba5-430b-a859-55fc674871da","Type":"ContainerDied","Data":"07cafb82d395ec533080a0e8418b5eb9d7d743c715642d4cabe3af2d42c3bf15"}
Jul 31 15:07:39.850354 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.849910    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"
Jul 31 15:07:39.850354 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.849959    2575 scope.go:117] "RemoveContainer" containerID="cba29875474cd344756e25f0e64e5a1f52eae7c8a098f53506d59fb4ad5d00bf"
Jul 31 15:07:39.858864 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.858838    2575 scope.go:117] "RemoveContainer" containerID="86811824a8a1f2e5c9607abadb3fc7c787e8343570ccb7b0405806c997f9cae1"
Jul 31 15:07:39.867370 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.867347    2575 scope.go:117] "RemoveContainer" containerID="16abc36233bec85a3211c8fa84277d5410eed327e9da933979abb5e2b8bdca07"
Jul 31 15:07:39.870430 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.870399    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:07:39.874920 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.874890    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-126e1-predictor-5d5f845887-hpvnv"]
Jul 31 15:07:39.877300 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:39.877282    2575 scope.go:117] "RemoveContainer" containerID="02acd25e1af93b1b76407ebac2b721a6aa9f81a3ed820d0cea186e412547943d"
Jul 31 15:07:41.202190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:41.202154    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc92568-1ba5-430b-a859-55fc674871da" path="/var/lib/kubelet/pods/1dc92568-1ba5-430b-a859-55fc674871da/volumes"
Jul 31 15:07:49.674968 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:49.674922    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:07:59.674959 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:07:59.674910    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:08:09.675289 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:09.675227    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:08:19.675808 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:19.675777    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:08:27.093434 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:27.093394    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:08:27.093962 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:27.093657    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" containerID="cri-o://ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741" gracePeriod=30
Jul 31 15:08:27.093962 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:27.093726    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kube-rbac-proxy" containerID="cri-o://6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7" gracePeriod=30
Jul 31 15:08:27.133350 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:27.133312    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:08:28.072102 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:28.072067    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerID="6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7" exitCode=2
Jul 31 15:08:28.072365 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:28.072143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerDied","Data":"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"}
Jul 31 15:08:29.668340 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:29.668298    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 31 15:08:29.675473 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:29.675427    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:08:32.219525 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.219491    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:08:32.533504 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.533398    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:08:32.557347 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.557316    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:08:32.612794 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.612760    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert\") pod \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") "
Jul 31 15:08:32.612996 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.612806    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8zwc\" (UniqueName: \"kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc\") pod \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") "
Jul 31 15:08:32.612996 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.612828    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location\") pod \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") "
Jul 31 15:08:32.612996 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.612870    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls\") pod \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") "
Jul 31 15:08:32.612996 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.612918    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config\") pod \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\" (UID: \"d2ae3d97-b27f-4234-a6c5-ed952ea65499\") "
Jul 31 15:08:32.613235 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.613185    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2ae3d97-b27f-4234-a6c5-ed952ea65499" (UID: "d2ae3d97-b27f-4234-a6c5-ed952ea65499"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:32.613305 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.613191    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2ae3d97-b27f-4234-a6c5-ed952ea65499" (UID: "d2ae3d97-b27f-4234-a6c5-ed952ea65499"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:08:32.613374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.613353    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2ae3d97-b27f-4234-a6c5-ed952ea65499" (UID: "d2ae3d97-b27f-4234-a6c5-ed952ea65499"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:08:32.615136 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.615110    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc" (OuterVolumeSpecName: "kube-api-access-h8zwc") pod "d2ae3d97-b27f-4234-a6c5-ed952ea65499" (UID: "d2ae3d97-b27f-4234-a6c5-ed952ea65499"). InnerVolumeSpecName "kube-api-access-h8zwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:08:32.615220 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.615197    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2ae3d97-b27f-4234-a6c5-ed952ea65499" (UID: "d2ae3d97-b27f-4234-a6c5-ed952ea65499"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:08:32.713845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.713805    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:08:32.713845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.713839    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2ae3d97-b27f-4234-a6c5-ed952ea65499-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:08:32.713845 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.713850    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zwc\" (UniqueName: \"kubernetes.io/projected/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kube-api-access-h8zwc\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:08:32.714072 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.713859    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ae3d97-b27f-4234-a6c5-ed952ea65499-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:08:32.714072 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:32.713870    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2ae3d97-b27f-4234-a6c5-ed952ea65499-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:08:33.096709 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.096672    2575 generic.go:356] "Generic (PLEG): container finished" podID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerID="ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741" exitCode=0
Jul 31 15:08:33.096917 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.096739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"
Jul 31 15:08:33.096917 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.096765    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerDied","Data":"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"}
Jul 31 15:08:33.096917 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.096840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5" event={"ID":"d2ae3d97-b27f-4234-a6c5-ed952ea65499","Type":"ContainerDied","Data":"d92045276fe70bd3492150716909ba9e70b0a06e4ae46315fa1e1f3d5b2554c7"}
Jul 31 15:08:33.096917 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.096890    2575 scope.go:117] "RemoveContainer" containerID="6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"
Jul 31 15:08:33.106505 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.106485    2575 scope.go:117] "RemoveContainer" containerID="ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"
Jul 31 15:08:33.116895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.116872    2575 scope.go:117] "RemoveContainer" containerID="827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"
Jul 31 15:08:33.121184 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.121153    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:08:33.126052 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.126031    2575 scope.go:117] "RemoveContainer" containerID="6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"
Jul 31 15:08:33.126477 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:08:33.126451    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7\": container with ID starting with 6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7 not found: ID does not exist" containerID="6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"
Jul 31 15:08:33.126675 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.126642    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7"} err="failed to get container status \"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7\": rpc error: code = NotFound desc = could not find container \"6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7\": container with ID starting with 6e93c770b184981d33153afa478a8bf0d2497d6105905ac5a5617ea468dd05d7 not found: ID does not exist"
Jul 31 15:08:33.126675 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.126677    2575 scope.go:117] "RemoveContainer" containerID="ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"
Jul 31 15:08:33.127019 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:08:33.127001    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741\": container with ID starting with ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741 not found: ID does not exist" containerID="ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"
Jul 31 15:08:33.127112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.127028    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741"} err="failed to get container status \"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741\": rpc error: code = NotFound desc = could not find container \"ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741\": container with ID starting with ae13556b4cfdf4c2ea3defd69dc5d1c61cdcb3525cadb42e2de3507e593de741 not found: ID does not exist"
Jul 31 15:08:33.127112 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.127047    2575 scope.go:117] "RemoveContainer" containerID="827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"
Jul 31 15:08:33.127330 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:08:33.127310    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d\": container with ID starting with 827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d not found: ID does not exist" containerID="827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"
Jul 31 15:08:33.127389 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.127341    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d"} err="failed to get container status \"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d\": rpc error: code = NotFound desc = could not find container \"827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d\": container with ID starting with 827b3958f1b08dda845a35bde859803c0450684fd08534bf456b5018471e168d not found: ID does not exist"
Jul 31 15:08:33.128403 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.128383    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-2d318-predictor-7bfc5bcb95-7snl5"]
Jul 31 15:08:33.202920 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:08:33.202886    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" path="/var/lib/kubelet/pods/d2ae3d97-b27f-4234-a6c5-ed952ea65499/volumes"
Jul 31 15:09:15.700341 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:15.700296    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:49.030480 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:49.030419    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:54.140737 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.140692    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:54.180057 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180009    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:09:54.180406 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180389    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kube-rbac-proxy"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180409    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kube-rbac-proxy"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180425    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180433    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180454    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180465    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180477    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="storage-initializer"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180487    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="storage-initializer"
Jul 31 15:09:54.180497 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180497    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="storage-initializer"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180506    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="storage-initializer"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180520    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180540    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180556    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180565    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180665    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180680    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kube-rbac-proxy"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180693    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="kserve-container"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180703    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc92568-1ba5-430b-a859-55fc674871da" containerName="agent"
Jul 31 15:09:54.180929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.180713    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ae3d97-b27f-4234-a6c5-ed952ea65499" containerName="kube-rbac-proxy"
Jul 31 15:09:54.184068 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.184040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.186677 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.186648    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 31 15:09:54.186835 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.186654    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 31 15:09:54.186937 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.186923    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mhdbw\""
Jul 31 15:09:54.187120 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.187094    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:09:54.187244 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.187127    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:09:54.187244 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.187141    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:09:54.187734 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.187718    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:09:54.193560 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.193535    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:09:54.220032 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.219996    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:54.223580 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.223545    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:09:54.227251 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.227220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.229737 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.229714    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 31 15:09:54.237778 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.237747    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:09:54.259965 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.259926    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d8s\" (UniqueName: \"kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.260156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.259976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.260156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.260039    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.260156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.260088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.260156 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.260114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.295637 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.295562    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:54.361517 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d8s\" (UniqueName: \"kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.361704 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.361704 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.361704 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.361704 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.361704 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361682    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:09:54.361731    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361745    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:09:54.361801    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls podName:e0f131c8-642e-47b9-b879-0128e6960a48 nodeName:}" failed. No retries permitted until 2026-07-31 15:09:54.861780724 +0000 UTC m=+1318.195900631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls") pod "isvc-canary-create-predictor-5cd5878b66-r6vgc" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwgz\" (UniqueName: \"kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.361990 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.361854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.362254 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.362094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.362292 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.362263    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.362340 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.362294    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.370853 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.370828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d8s\" (UniqueName: \"kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.462625 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.462528    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.462625 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.462563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.462625 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.462580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.462928 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.462639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwgz\" (UniqueName: \"kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.462928 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.462663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.463095 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.463071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.463273 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.463256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.463360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.463342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.465353 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.465337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.472430 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.472398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwgz\" (UniqueName: \"kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz\") pod \"isvc-canary-create-v2-predictor-5f99649479-jdxl4\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.508315 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.508277    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:09:54.538654 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.538584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:54.687868 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.687841    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:09:54.690417 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:09:54.690379    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded455a86_6b6a_473c_944b_e102de85d010.slice/crio-6c55afd58fc8d64a8eb39012020ef6422617d5a045ac6376f7c1f857343ced84 WatchSource:0}: Error finding container 6c55afd58fc8d64a8eb39012020ef6422617d5a045ac6376f7c1f857343ced84: Status 404 returned error can't find the container with id 6c55afd58fc8d64a8eb39012020ef6422617d5a045ac6376f7c1f857343ced84
Jul 31 15:09:54.867444 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.867409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:54.870085 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:54.870049    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-r6vgc\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:55.096645 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.096514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:09:55.244239 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.244203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:09:55.247572 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:09:55.247541    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f131c8_642e_47b9_b879_0128e6960a48.slice/crio-16d07ab8dc60c91b7e64217b76e43bc75b3819682882a7ea74737dcf5a5175c0 WatchSource:0}: Error finding container 16d07ab8dc60c91b7e64217b76e43bc75b3819682882a7ea74737dcf5a5175c0: Status 404 returned error can't find the container with id 16d07ab8dc60c91b7e64217b76e43bc75b3819682882a7ea74737dcf5a5175c0
Jul 31 15:09:55.468040 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.467992    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerStarted","Data":"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"}
Jul 31 15:09:55.468250 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.468120    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerStarted","Data":"16d07ab8dc60c91b7e64217b76e43bc75b3819682882a7ea74737dcf5a5175c0"}
Jul 31 15:09:55.469755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.469727    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerStarted","Data":"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"}
Jul 31 15:09:55.469879 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:55.469779    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerStarted","Data":"6c55afd58fc8d64a8eb39012020ef6422617d5a045ac6376f7c1f857343ced84"}
Jul 31 15:09:58.485541 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:58.485501    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed455a86-6b6a-473c-944b-e102de85d010" containerID="e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894" exitCode=0
Jul 31 15:09:58.485987 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:58.485581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerDied","Data":"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"}
Jul 31 15:09:59.500491 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.500441    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0f131c8-642e-47b9-b879-0128e6960a48" containerID="cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b" exitCode=0
Jul 31 15:09:59.501031 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.500518    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerDied","Data":"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"}
Jul 31 15:09:59.502890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.502865    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerStarted","Data":"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"}
Jul 31 15:09:59.502997 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.502925    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerStarted","Data":"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"}
Jul 31 15:09:59.503144 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.503126    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:09:59.544080 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:09:59.544018    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podStartSLOduration=1.870367224 podStartE2EDuration="5.543999253s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.673632029s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:59.542937176 +0000 UTC m=+1322.877057114" watchObservedRunningTime="2026-07-31 15:09:59.543999253 +0000 UTC m=+1322.878119174"
Jul 31 15:10:00.509481 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:00.509441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerStarted","Data":"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"}
Jul 31 15:10:00.509914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:00.509510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerStarted","Data":"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"}
Jul 31 15:10:00.509914 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:00.509791    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:10:00.511232 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:00.511204    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:00.531974 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:00.531923    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podStartSLOduration=2.950823153 podStartE2EDuration="6.531906233s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.58108308s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:00.530774606 +0000 UTC m=+1323.864894527" watchObservedRunningTime="2026-07-31 15:10:00.531906233 +0000 UTC m=+1323.866026173"
Jul 31 15:10:01.514509 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:01.514468    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:02.355164 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:02.355102    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:10:05.510685 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:05.510636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:10:05.511189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:05.511168    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:10:05.512365 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:05.512337    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:05.516814 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:05.516780    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:10:05.535107 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:05.535055    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:06.520309 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:06.520282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:10:06.521015 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:06.520984    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:06.539562 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:06.539523    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:16.521493 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:16.521449    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:16.540316 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:16.540270    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:26.520981 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:26.520887    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:26.540433 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:26.540382    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:36.520992 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:36.520945    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:36.540357 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:36.540313    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:46.521472 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:46.521417    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:46.539999 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:46.539944    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:10:56.521043 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:56.520988    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:10:56.540360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:10:56.540313    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:11:06.521637 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:06.521575    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:11:06.539753 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:06.539715    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:11:07.199192 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:07.199154    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:11:16.539697 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:16.539648    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:11:17.201882 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:17.201850    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:11:17.202071 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:17.201947    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:11:24.300669 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.300624    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:11:24.301127 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.300889    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" containerID="cri-o://6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36" gracePeriod=30
Jul 31 15:11:24.301127 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.300949    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kube-rbac-proxy" containerID="cri-o://b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd" gracePeriod=30
Jul 31 15:11:24.307423 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.306857    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:11:24.307423 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.307173    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" containerID="cri-o://00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f" gracePeriod=30
Jul 31 15:11:24.307423 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.307314    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kube-rbac-proxy" containerID="cri-o://78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c" gracePeriod=30
Jul 31 15:11:24.316921 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.316888    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:24.397557 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.397525    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:24.898201 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.898163    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0f131c8-642e-47b9-b879-0128e6960a48" containerID="b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd" exitCode=2
Jul 31 15:11:24.898379 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.898241    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerDied","Data":"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"}
Jul 31 15:11:24.900405 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.900375    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed455a86-6b6a-473c-944b-e102de85d010" containerID="78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c" exitCode=2
Jul 31 15:11:24.900553 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:24.900433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerDied","Data":"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"}
Jul 31 15:11:25.517879 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:25.517833    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 31 15:11:26.515105 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:26.515057    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 31 15:11:27.198822 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:27.198771    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:11:27.199275 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:27.199055    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:11:29.416481 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.416447    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:29.494437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.494399    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:11:29.499577 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.499548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.502164 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.502139    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 31 15:11:29.502304 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.502249    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 31 15:11:29.507028 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.506662    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:29.515114 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.515076    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:11:29.581661 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.581587    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxdxl\" (UniqueName: \"kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.581856 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.581674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.581856 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.581706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.581856 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.581729    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.582035 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.581847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.620019 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.619986    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:29.682317 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.682490 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.682490 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.682490 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.682490 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxdxl\" (UniqueName: \"kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.682780 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.682519    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 31 15:11:29.682780 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.682629    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls podName:e4b8c1d0-5bfa-4fb7-9130-a53b95324920 nodeName:}" failed. No retries permitted until 2026-07-31 15:11:30.182574474 +0000 UTC m=+1413.516694374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls") pod "isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Jul 31 15:11:29.682780 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.682767    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.683236 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.683204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.683334 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.683297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.692807 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.692761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxdxl\" (UniqueName: \"kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:29.748249 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.748214    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:11:29.777702 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.777662    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:11:29.783021 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.782996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") pod \"e0f131c8-642e-47b9-b879-0128e6960a48\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") "
Jul 31 15:11:29.783161 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57d8s\" (UniqueName: \"kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s\") pod \"e0f131c8-642e-47b9-b879-0128e6960a48\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") "
Jul 31 15:11:29.783161 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783101    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert\") pod \"e0f131c8-642e-47b9-b879-0128e6960a48\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") "
Jul 31 15:11:29.783161 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783126    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config\") pod \"e0f131c8-642e-47b9-b879-0128e6960a48\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") "
Jul 31 15:11:29.783373 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783175    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location\") pod \"e0f131c8-642e-47b9-b879-0128e6960a48\" (UID: \"e0f131c8-642e-47b9-b879-0128e6960a48\") "
Jul 31 15:11:29.783639 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783571    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0f131c8-642e-47b9-b879-0128e6960a48" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:29.783756 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783724    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0f131c8-642e-47b9-b879-0128e6960a48" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:29.783978 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.783853    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0f131c8-642e-47b9-b879-0128e6960a48" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:29.787259 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.787217    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s" (OuterVolumeSpecName: "kube-api-access-57d8s") pod "e0f131c8-642e-47b9-b879-0128e6960a48" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48"). InnerVolumeSpecName "kube-api-access-57d8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:29.787421 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.787271    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0f131c8-642e-47b9-b879-0128e6960a48" (UID: "e0f131c8-642e-47b9-b879-0128e6960a48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:29.867317 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.867286    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:11:29.884623 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884567    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls\") pod \"ed455a86-6b6a-473c-944b-e102de85d010\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") "
Jul 31 15:11:29.884829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884667    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location\") pod \"ed455a86-6b6a-473c-944b-e102de85d010\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") "
Jul 31 15:11:29.884829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884701    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dwgz\" (UniqueName: \"kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz\") pod \"ed455a86-6b6a-473c-944b-e102de85d010\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") "
Jul 31 15:11:29.884829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884753    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert\") pod \"ed455a86-6b6a-473c-944b-e102de85d010\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") "
Jul 31 15:11:29.884829 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config\") pod \"ed455a86-6b6a-473c-944b-e102de85d010\" (UID: \"ed455a86-6b6a-473c-944b-e102de85d010\") "
Jul 31 15:11:29.885047 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.884992    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed455a86-6b6a-473c-944b-e102de85d010" (UID: "ed455a86-6b6a-473c-944b-e102de85d010"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:29.885106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885044    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.885106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885062    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f131c8-642e-47b9-b879-0128e6960a48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.885106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885081    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f131c8-642e-47b9-b879-0128e6960a48-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.885106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885098    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f131c8-642e-47b9-b879-0128e6960a48-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.885251 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885112    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57d8s\" (UniqueName: \"kubernetes.io/projected/e0f131c8-642e-47b9-b879-0128e6960a48-kube-api-access-57d8s\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.885251 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885152    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed455a86-6b6a-473c-944b-e102de85d010" (UID: "ed455a86-6b6a-473c-944b-e102de85d010"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:29.885251 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.885182    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed455a86-6b6a-473c-944b-e102de85d010" (UID: "ed455a86-6b6a-473c-944b-e102de85d010"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:29.887012 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.886984    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed455a86-6b6a-473c-944b-e102de85d010" (UID: "ed455a86-6b6a-473c-944b-e102de85d010"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:29.887070 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.887022    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz" (OuterVolumeSpecName: "kube-api-access-4dwgz") pod "ed455a86-6b6a-473c-944b-e102de85d010" (UID: "ed455a86-6b6a-473c-944b-e102de85d010"). InnerVolumeSpecName "kube-api-access-4dwgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:29.925956 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.925853    2575 generic.go:356] "Generic (PLEG): container finished" podID="e0f131c8-642e-47b9-b879-0128e6960a48" containerID="6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36" exitCode=0
Jul 31 15:11:29.925956 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.925920    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"
Jul 31 15:11:29.925956 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.925926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerDied","Data":"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"}
Jul 31 15:11:29.926233 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.926000    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc" event={"ID":"e0f131c8-642e-47b9-b879-0128e6960a48","Type":"ContainerDied","Data":"16d07ab8dc60c91b7e64217b76e43bc75b3819682882a7ea74737dcf5a5175c0"}
Jul 31 15:11:29.926233 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.926041    2575 scope.go:117] "RemoveContainer" containerID="b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"
Jul 31 15:11:29.928163 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.928135    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed455a86-6b6a-473c-944b-e102de85d010" containerID="00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f" exitCode=0
Jul 31 15:11:29.928304 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.928252    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerDied","Data":"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"}
Jul 31 15:11:29.928304 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.928273    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"
Jul 31 15:11:29.928392 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.928315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4" event={"ID":"ed455a86-6b6a-473c-944b-e102de85d010","Type":"ContainerDied","Data":"6c55afd58fc8d64a8eb39012020ef6422617d5a045ac6376f7c1f857343ced84"}
Jul 31 15:11:29.936796 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.936584    2575 scope.go:117] "RemoveContainer" containerID="6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"
Jul 31 15:11:29.945684 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.945650    2575 scope.go:117] "RemoveContainer" containerID="cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"
Jul 31 15:11:29.955035 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.955002    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:11:29.955654 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.955637    2575 scope.go:117] "RemoveContainer" containerID="b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"
Jul 31 15:11:29.956020 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.955997    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd\": container with ID starting with b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd not found: ID does not exist" containerID="b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"
Jul 31 15:11:29.956129 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956029    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd"} err="failed to get container status \"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd\": rpc error: code = NotFound desc = could not find container \"b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd\": container with ID starting with b152c23236c90d1a3dc30414f0ead0a7c64be5242f14c1bf8630c7383de61bbd not found: ID does not exist"
Jul 31 15:11:29.956129 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956051    2575 scope.go:117] "RemoveContainer" containerID="6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"
Jul 31 15:11:29.956384 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.956362    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36\": container with ID starting with 6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36 not found: ID does not exist" containerID="6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"
Jul 31 15:11:29.956478 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956384    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36"} err="failed to get container status \"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36\": rpc error: code = NotFound desc = could not find container \"6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36\": container with ID starting with 6053386997b37aeb8c7a0cf4126b4084665f1c02f1762e29eecf1ae6f0507a36 not found: ID does not exist"
Jul 31 15:11:29.956478 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956403    2575 scope.go:117] "RemoveContainer" containerID="cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"
Jul 31 15:11:29.957124 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.956742    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b\": container with ID starting with cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b not found: ID does not exist" containerID="cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"
Jul 31 15:11:29.957124 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956775    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b"} err="failed to get container status \"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b\": rpc error: code = NotFound desc = could not find container \"cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b\": container with ID starting with cf2e450716d4f4f0dd594abd7cf2cf24ca1046cd6f1c471fa9fc4ac9b309395b not found: ID does not exist"
Jul 31 15:11:29.957124 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.956795    2575 scope.go:117] "RemoveContainer" containerID="78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"
Jul 31 15:11:29.959424 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.959398    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-jdxl4"]
Jul 31 15:11:29.965729 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.965703    2575 scope.go:117] "RemoveContainer" containerID="00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"
Jul 31 15:11:29.971310 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.971277    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:11:29.975005 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.974981    2575 scope.go:117] "RemoveContainer" containerID="e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"
Jul 31 15:11:29.975731 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.975708    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-r6vgc"]
Jul 31 15:11:29.983718 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.983697    2575 scope.go:117] "RemoveContainer" containerID="78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"
Jul 31 15:11:29.984073 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.984047    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c\": container with ID starting with 78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c not found: ID does not exist" containerID="78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"
Jul 31 15:11:29.984189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.984081    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c"} err="failed to get container status \"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c\": rpc error: code = NotFound desc = could not find container \"78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c\": container with ID starting with 78666056967e3e26dc4b82618056598ccc149ca9c9796f5d46194389716a480c not found: ID does not exist"
Jul 31 15:11:29.984189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.984101    2575 scope.go:117] "RemoveContainer" containerID="00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"
Jul 31 15:11:29.984374 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.984355    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f\": container with ID starting with 00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f not found: ID does not exist" containerID="00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"
Jul 31 15:11:29.984441 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.984386    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f"} err="failed to get container status \"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f\": rpc error: code = NotFound desc = could not find container \"00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f\": container with ID starting with 00725bc9eca9e9b6df1475cea4bc3d402571057034ced4f6129de4543a74f89f not found: ID does not exist"
Jul 31 15:11:29.984441 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.984414    2575 scope.go:117] "RemoveContainer" containerID="e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"
Jul 31 15:11:29.984727 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:11:29.984703    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894\": container with ID starting with e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894 not found: ID does not exist" containerID="e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"
Jul 31 15:11:29.984792 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.984736    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894"} err="failed to get container status \"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894\": rpc error: code = NotFound desc = could not find container \"e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894\": container with ID starting with e1bdda35722d926709d3af46b902c589c68f241ae1b5074674045d8e60557894 not found: ID does not exist"
Jul 31 15:11:29.986096 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.986073    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed455a86-6b6a-473c-944b-e102de85d010-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.986172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.986098    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed455a86-6b6a-473c-944b-e102de85d010-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.986172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.986110    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwgz\" (UniqueName: \"kubernetes.io/projected/ed455a86-6b6a-473c-944b-e102de85d010-kube-api-access-4dwgz\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.986172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.986118    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:29.986172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:29.986127    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed455a86-6b6a-473c-944b-e102de85d010-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:11:30.188121 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.188017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:30.190917 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.190892    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:30.410468 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.410436    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:30.561210 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.561185    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:11:30.563288 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:11:30.563259    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b8c1d0_5bfa_4fb7_9130_a53b95324920.slice/crio-950a029699b8297504b5cd548f1a0cf0c185fcadf0f70f9e97fe959439e910bb WatchSource:0}: Error finding container 950a029699b8297504b5cd548f1a0cf0c185fcadf0f70f9e97fe959439e910bb: Status 404 returned error can't find the container with id 950a029699b8297504b5cd548f1a0cf0c185fcadf0f70f9e97fe959439e910bb
Jul 31 15:11:30.934172 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.934131    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerStarted","Data":"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"}
Jul 31 15:11:30.934403 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:30.934207    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerStarted","Data":"950a029699b8297504b5cd548f1a0cf0c185fcadf0f70f9e97fe959439e910bb"}
Jul 31 15:11:31.202700 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:31.202580    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" path="/var/lib/kubelet/pods/e0f131c8-642e-47b9-b879-0128e6960a48/volumes"
Jul 31 15:11:31.203100 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:31.203084    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed455a86-6b6a-473c-944b-e102de85d010" path="/var/lib/kubelet/pods/ed455a86-6b6a-473c-944b-e102de85d010/volumes"
Jul 31 15:11:34.956401 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:34.956364    2575 generic.go:356] "Generic (PLEG): container finished" podID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerID="440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c" exitCode=0
Jul 31 15:11:34.956941 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:34.956449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerDied","Data":"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"}
Jul 31 15:11:35.963648 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.963592    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerStarted","Data":"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"}
Jul 31 15:11:35.964052 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.963679    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerStarted","Data":"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"}
Jul 31 15:11:35.964052 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.963958    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:35.964130 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.964096    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:35.965424 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.965395    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:11:35.984361 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:35.984305    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podStartSLOduration=3.345736978 podStartE2EDuration="6.984284763s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.638547785s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:35.983415137 +0000 UTC m=+1419.317535059" watchObservedRunningTime="2026-07-31 15:11:35.984284763 +0000 UTC m=+1419.318404772"
Jul 31 15:11:36.968255 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:36.968213    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:11:41.973068 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:41.973034    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:11:41.973751 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:41.973720    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:11:51.974626 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:11:51.974514    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:01.973816 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:01.973767    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:11.973884 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:11.973841    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:21.974102 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:21.974048    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:31.974327 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:31.974281    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:41.974440 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:41.974392    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:12:51.974444 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:12:51.974403    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:13:01.974371 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:13:01.974335    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:13:05.478823 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:13:05.478780    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:13:49.596441 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:13:49.596409    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:14:34.854543 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:34.854504    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:14:34.855929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:34.854801    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container" containerID="cri-o://de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d" gracePeriod=30
Jul 31 15:14:34.855929 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:34.854865    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kube-rbac-proxy" containerID="cri-o://a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828" gracePeriod=30
Jul 31 15:14:34.884221 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:34.884184    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:14:35.779554 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:35.779519    2575 generic.go:356] "Generic (PLEG): container finished" podID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerID="a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828" exitCode=2
Jul 31 15:14:35.779764 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:35.779612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerDied","Data":"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"}
Jul 31 15:14:36.969147 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:36.969105    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 31 15:14:40.316617 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.316568    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:14:40.346715 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.346675    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxdxl\" (UniqueName: \"kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl\") pod \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") "
Jul 31 15:14:40.346931 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.346761    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert\") pod \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") "
Jul 31 15:14:40.346931 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.346858    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location\") pod \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") "
Jul 31 15:14:40.346931 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.346909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config\") pod \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") "
Jul 31 15:14:40.347086 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.346969    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") pod \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\" (UID: \"e4b8c1d0-5bfa-4fb7-9130-a53b95324920\") "
Jul 31 15:14:40.347246 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.347186    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4b8c1d0-5bfa-4fb7-9130-a53b95324920" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:40.347510 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.347475    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4b8c1d0-5bfa-4fb7-9130-a53b95324920" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:40.347702 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.347675    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4b8c1d0-5bfa-4fb7-9130-a53b95324920" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:40.350223 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.350185    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4b8c1d0-5bfa-4fb7-9130-a53b95324920" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:40.350223 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.350197    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl" (OuterVolumeSpecName: "kube-api-access-cxdxl") pod "e4b8c1d0-5bfa-4fb7-9130-a53b95324920" (UID: "e4b8c1d0-5bfa-4fb7-9130-a53b95324920"). InnerVolumeSpecName "kube-api-access-cxdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:40.448009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.447910    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxdxl\" (UniqueName: \"kubernetes.io/projected/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-api-access-cxdxl\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.448009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.447947    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.448009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.447958    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.448009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.447967    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.448009 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.447976    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b8c1d0-5bfa-4fb7-9130-a53b95324920-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:14:40.807111 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.807027    2575 generic.go:356] "Generic (PLEG): container finished" podID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerID="de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d" exitCode=0
Jul 31 15:14:40.807111 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.807089    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerDied","Data":"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"}
Jul 31 15:14:40.807111 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.807095    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"
Jul 31 15:14:40.807326 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.807148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd" event={"ID":"e4b8c1d0-5bfa-4fb7-9130-a53b95324920","Type":"ContainerDied","Data":"950a029699b8297504b5cd548f1a0cf0c185fcadf0f70f9e97fe959439e910bb"}
Jul 31 15:14:40.807326 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.807179    2575 scope.go:117] "RemoveContainer" containerID="a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"
Jul 31 15:14:40.816519 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.816499    2575 scope.go:117] "RemoveContainer" containerID="de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"
Jul 31 15:14:40.825199 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.825165    2575 scope.go:117] "RemoveContainer" containerID="440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"
Jul 31 15:14:40.831292 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.831263    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:14:40.834189 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.834158    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-l4pcd"]
Jul 31 15:14:40.834763 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.834746    2575 scope.go:117] "RemoveContainer" containerID="a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"
Jul 31 15:14:40.835148 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:14:40.835127    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828\": container with ID starting with a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828 not found: ID does not exist" containerID="a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"
Jul 31 15:14:40.835193 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.835162    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828"} err="failed to get container status \"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828\": rpc error: code = NotFound desc = could not find container \"a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828\": container with ID starting with a6a611abc83c5c90f47b61e6a735e06de7e82c5996e7c004f76a92ac4df1b828 not found: ID does not exist"
Jul 31 15:14:40.835193 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.835181    2575 scope.go:117] "RemoveContainer" containerID="de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"
Jul 31 15:14:40.835474 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:14:40.835454    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d\": container with ID starting with de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d not found: ID does not exist" containerID="de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"
Jul 31 15:14:40.835518 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.835481    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d"} err="failed to get container status \"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d\": rpc error: code = NotFound desc = could not find container \"de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d\": container with ID starting with de6eca5cfa49f336fdae7f599e410794125e0f7627e09d19cd74ca1113165f1d not found: ID does not exist"
Jul 31 15:14:40.835518 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.835498    2575 scope.go:117] "RemoveContainer" containerID="440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"
Jul 31 15:14:40.835798 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:14:40.835779    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c\": container with ID starting with 440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c not found: ID does not exist" containerID="440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"
Jul 31 15:14:40.835847 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:40.835801    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c"} err="failed to get container status \"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c\": rpc error: code = NotFound desc = could not find container \"440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c\": container with ID starting with 440beb80f65c6bb664f358f54d6fe2a030ba6963376f09f4ef1d8a9e2f5bc88c not found: ID does not exist"
Jul 31 15:14:41.201512 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:41.201470    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" path="/var/lib/kubelet/pods/e4b8c1d0-5bfa-4fb7-9130-a53b95324920/volumes"
Jul 31 15:14:48.659436 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:48.659337    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:14:48.789041 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:48.789004    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:14:58.756520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.756484    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:14:58.801765 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.801729    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:14:58.802048 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802036    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802050    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802062    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802067    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802077    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="storage-initializer"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802083    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="storage-initializer"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802089    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802096    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802103    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="storage-initializer"
Jul 31 15:14:58.802108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802108    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="storage-initializer"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802118    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802123    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802131    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802135    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802145    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="storage-initializer"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802150    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="storage-initializer"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802156    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802161    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802224    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802234    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802240    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f131c8-642e-47b9-b879-0128e6960a48" containerName="kserve-container"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802245    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802251    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed455a86-6b6a-473c-944b-e102de85d010" containerName="kube-rbac-proxy"
Jul 31 15:14:58.802374 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.802258    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b8c1d0-5bfa-4fb7-9130-a53b95324920" containerName="kserve-container"
Jul 31 15:14:58.805294 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.805272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:58.808546 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.808519    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Jul 31 15:14:58.808546 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.808526    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:14:58.808813 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.808793    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:14:58.809415 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.809399    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mhdbw\""
Jul 31 15:14:58.809479 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.809462    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Jul 31 15:14:58.809538 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.809485    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:14:58.809817 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.809803    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:14:58.824795 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.820924    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:14:58.905091 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.905037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4n7\" (UniqueName: \"kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:58.905091 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.905094    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:58.905332 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.905206    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:58.905332 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.905274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:58.905332 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:58.905313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.005772 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.005737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4n7\" (UniqueName: \"kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.005772 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.005777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006012 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.005812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006012 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.005847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006012 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:14:59.005939    2575 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Jul 31 15:14:59.006012 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:14:59.006002    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls podName:1e91ba2c-b1db-4c86-bc0d-6734a9c822ff nodeName:}" failed. No retries permitted until 2026-07-31 15:14:59.505983702 +0000 UTC m=+1622.840103602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls") pod "isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Jul 31 15:14:59.006215 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.006032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006385 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.006359    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006495 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.006476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.006582 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.006538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.014543 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.014513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4n7\" (UniqueName: \"kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.510896 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.510847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.513752 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.513715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.716242 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.716196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:14:59.864481 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.864453    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:14:59.866634 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:14:59.866574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e91ba2c_b1db_4c86_bc0d_6734a9c822ff.slice/crio-3b41fbec2d127fd55486d7f5ffd61186eb26cc99267596bf5f4fa4caf963bf91 WatchSource:0}: Error finding container 3b41fbec2d127fd55486d7f5ffd61186eb26cc99267596bf5f4fa4caf963bf91: Status 404 returned error can't find the container with id 3b41fbec2d127fd55486d7f5ffd61186eb26cc99267596bf5f4fa4caf963bf91
Jul 31 15:14:59.891337 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:14:59.891300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerStarted","Data":"3b41fbec2d127fd55486d7f5ffd61186eb26cc99267596bf5f4fa4caf963bf91"}
Jul 31 15:15:00.896977 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:00.896935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerStarted","Data":"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"}
Jul 31 15:15:00.919163 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:00.919117    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:15:04.916451 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:04.916406    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerID="eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d" exitCode=0
Jul 31 15:15:04.916902 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:04.916463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerDied","Data":"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"}
Jul 31 15:15:05.923464 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.923422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerStarted","Data":"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"}
Jul 31 15:15:05.923895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.923496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerStarted","Data":"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"}
Jul 31 15:15:05.923895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.923521    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerStarted","Data":"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"}
Jul 31 15:15:05.923895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.923822    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:15:05.923895 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.923885    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:15:05.925417 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.925385    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:05.949558 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:05.949496    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podStartSLOduration=3.916158985 podStartE2EDuration="7.949477739s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.033318754s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:05.94707354 +0000 UTC m=+1629.281193476" watchObservedRunningTime="2026-07-31 15:15:05.949477739 +0000 UTC m=+1629.283597661"
Jul 31 15:15:06.929086 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:06.929039    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:15:06.929647 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:06.929212    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:06.930360 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:06.930329    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:07.934170 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:07.934118    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:07.934589 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:07.934566    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:12.939169 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:12.939138    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:15:12.939806 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:12.939774    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:12.940117 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:12.940098    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:22.940149 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:22.940101    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:22.940740 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:22.940577    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:32.940273 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:32.940224    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:32.940837 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:32.940813    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:42.939733 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:42.939682    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:42.940331 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:42.940174    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:15:52.940320 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:52.940265    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:15:52.940881 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:15:52.940703    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:16:02.940524 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:02.940471    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:16:02.941072 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:02.941028    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:16:12.940680 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:12.940626    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:16:12.941233 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:12.941205    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:16:22.940384 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:22.940290    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:16:22.940822 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:22.940700    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:16:33.995017 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:33.994974    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:16:33.995483 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:33.995250    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" containerID="cri-o://79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f" gracePeriod=30
Jul 31 15:16:33.995483 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:33.995269    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" containerID="cri-o://e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc" gracePeriod=30
Jul 31 15:16:33.995483 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:33.995333    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" containerID="cri-o://87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5" gracePeriod=30
Jul 31 15:16:34.057427 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:34.057383    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:16:34.115282 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:34.115251    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:16:34.326866 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:34.326779    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerID="87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5" exitCode=2
Jul 31 15:16:34.326866 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:34.326839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerDied","Data":"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"}
Jul 31 15:16:35.070876 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:35.070834    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:16:35.171058 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:35.171020    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:16:37.935092 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:37.935042    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:16:39.350988 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:39.350952    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerID="79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f" exitCode=0
Jul 31 15:16:39.351370 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:39.351027    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerDied","Data":"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"}
Jul 31 15:16:42.934887 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:42.934841    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:16:42.940329 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:42.940277    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:16:42.940635 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:42.940575    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:16:47.688272 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:47.688231    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:16:47.935095 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:47.935041    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:16:47.935291 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:47.935221    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:16:50.790905 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:50.790866    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:50.817468 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:50.817427    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:50.829290 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:50.829257    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:50.848771 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:50.848746    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:51.649331 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:51.649303    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:51.662720 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:51.662685    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:51.673562 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:51.673528    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:51.685521 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:51.685499    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:52.516968 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.516940    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:52.533298 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.533271    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:52.545245 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.545213    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:52.557496 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.557472    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:52.934860 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.934818    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:16:52.940414 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.940379    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:16:52.940792 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:52.940773    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:16:53.427124 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:53.427089    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:53.441825 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:53.441788    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:53.466106 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:53.466079    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:53.480944 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:53.480918    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:54.311160 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:54.311126    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:54.328343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:54.328311    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:54.341726 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:54.341696    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:54.354542 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:54.354517    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:55.153908 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:55.153881    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:55.168270 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:55.168235    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:55.180238 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:55.180206    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:55.196331 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:55.196300    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:56.072899 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:56.072860    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:56.096372 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:56.096341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:56.121492 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:56.121464    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:56.150128 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:56.150095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:57.029320 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.029285    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:57.051619 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.051577    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:57.068280 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.068248    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:57.092482 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.092456    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:57.909151 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.909117    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:57.924079 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.924046    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:57.935324 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.935278    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:16:57.936418 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.936396    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:57.950521 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:57.950496    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:58.788670 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:58.788638    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:58.809768 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:58.809738    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:58.823753 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:58.823719    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:58.836843 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:58.836814    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:16:59.637110 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:59.637084    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:16:59.651728 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:59.651696    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:16:59.665086 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:59.665056    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:16:59.680456 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:16:59.680427    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:17:00.513672 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:00.513646    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:17:00.527640 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:00.527613    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:17:00.539295 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:00.539257    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:17:00.553870 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:00.553840    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:17:01.367876 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:01.367851    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:17:01.393523 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:01.393487    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:17:01.419743 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:01.419718    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:17:01.443655 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:01.443621    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:17:02.332566 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.332538    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kserve-container/0.log"
Jul 31 15:17:02.352710 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.352680    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/kube-rbac-proxy/0.log"
Jul 31 15:17:02.369472 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.369436    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/agent/0.log"
Jul 31 15:17:02.384735 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.384706    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp_1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/storage-initializer/0.log"
Jul 31 15:17:02.934881 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.934834    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:17:02.940399 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.940363    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:17:02.940553 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.940538    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:17:02.940767 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.940743    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:17:02.940906 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:02.940838    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:17:04.146263 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.146237    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:17:04.248528 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248482    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location\") pod \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") "
Jul 31 15:17:04.248755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248543    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") pod \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") "
Jul 31 15:17:04.248755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert\") pod \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") "
Jul 31 15:17:04.248755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248588    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config\") pod \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") "
Jul 31 15:17:04.248755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248669    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd4n7\" (UniqueName: \"kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7\") pod \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\" (UID: \"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff\") "
Jul 31 15:17:04.248965 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248939    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:04.249025 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.248994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:04.249025 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.249004    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:04.250997 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.250921    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:17:04.251132 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.251105    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7" (OuterVolumeSpecName: "kube-api-access-sd4n7") pod "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" (UID: "1e91ba2c-b1db-4c86-bc0d-6734a9c822ff"). InnerVolumeSpecName "kube-api-access-sd4n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:04.349314 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.349268    2575 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:17:04.349314 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.349304    2575 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:17:04.349314 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.349314    2575 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:17:04.349314 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.349322    2575 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:17:04.349590 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.349331    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4n7\" (UniqueName: \"kubernetes.io/projected/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff-kube-api-access-sd4n7\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 31 15:17:04.473010 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.472973    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerID="e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc" exitCode=0
Jul 31 15:17:04.473190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.473043    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"
Jul 31 15:17:04.473190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.473052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerDied","Data":"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"}
Jul 31 15:17:04.473190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.473115    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp" event={"ID":"1e91ba2c-b1db-4c86-bc0d-6734a9c822ff","Type":"ContainerDied","Data":"3b41fbec2d127fd55486d7f5ffd61186eb26cc99267596bf5f4fa4caf963bf91"}
Jul 31 15:17:04.473190 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.473147    2575 scope.go:117] "RemoveContainer" containerID="e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"
Jul 31 15:17:04.483231 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.483212    2575 scope.go:117] "RemoveContainer" containerID="87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"
Jul 31 15:17:04.491848 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.491821    2575 scope.go:117] "RemoveContainer" containerID="79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"
Jul 31 15:17:04.497478 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.497447    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:17:04.501680 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.501630    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c7ccfc486-m7vwp"]
Jul 31 15:17:04.501680 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.501671    2575 scope.go:117] "RemoveContainer" containerID="eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"
Jul 31 15:17:04.510151 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.510130    2575 scope.go:117] "RemoveContainer" containerID="e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"
Jul 31 15:17:04.510500 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:17:04.510475    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc\": container with ID starting with e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc not found: ID does not exist" containerID="e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"
Jul 31 15:17:04.510550 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.510512    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc"} err="failed to get container status \"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc\": rpc error: code = NotFound desc = could not find container \"e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc\": container with ID starting with e380e2e4d8a79757a1f84971502ad9e8334d8172cf37a9df02b13794e97fc9dc not found: ID does not exist"
Jul 31 15:17:04.510550 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.510532    2575 scope.go:117] "RemoveContainer" containerID="87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"
Jul 31 15:17:04.510841 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:17:04.510821    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5\": container with ID starting with 87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5 not found: ID does not exist" containerID="87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"
Jul 31 15:17:04.510890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.510849    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5"} err="failed to get container status \"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5\": rpc error: code = NotFound desc = could not find container \"87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5\": container with ID starting with 87afa9b6a805e06e4a5fb4f57e4cce76d2023199d829b60abf8efcdaf4233db5 not found: ID does not exist"
Jul 31 15:17:04.510890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.510865    2575 scope.go:117] "RemoveContainer" containerID="79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"
Jul 31 15:17:04.511136 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:17:04.511114    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f\": container with ID starting with 79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f not found: ID does not exist" containerID="79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"
Jul 31 15:17:04.511188 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.511149    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f"} err="failed to get container status \"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f\": rpc error: code = NotFound desc = could not find container \"79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f\": container with ID starting with 79cf18bc4903869dc6f21d5969b2e6b377f4e482d0bb1f7a791f5fbfe793773f not found: ID does not exist"
Jul 31 15:17:04.511188 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.511168    2575 scope.go:117] "RemoveContainer" containerID="eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"
Jul 31 15:17:04.511418 ip-10-0-143-227 kubenswrapper[2575]: E0731 15:17:04.511401    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d\": container with ID starting with eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d not found: ID does not exist" containerID="eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"
Jul 31 15:17:04.511477 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:04.511420    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d"} err="failed to get container status \"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d\": rpc error: code = NotFound desc = could not find container \"eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d\": container with ID starting with eef8f34449ed5dc550570b68fd9bc90d46391c6184ec7432b87390f3f344b13d not found: ID does not exist"
Jul 31 15:17:05.201514 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:05.201474    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" path="/var/lib/kubelet/pods/1e91ba2c-b1db-4c86-bc0d-6734a9c822ff/volumes"
Jul 31 15:17:06.420361 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.420280    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:17:06.522506 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522467    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4zszp/must-gather-b9kgh"]
Jul 31 15:17:06.522821 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522806    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522823    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522835    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="storage-initializer"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522841    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="storage-initializer"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522848    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522855    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522873    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy"
Jul 31 15:17:06.522890 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522879    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy"
Jul 31 15:17:06.523108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522933    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kserve-container"
Jul 31 15:17:06.523108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522940    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="kube-rbac-proxy"
Jul 31 15:17:06.523108 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.522947    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e91ba2c-b1db-4c86-bc0d-6734a9c822ff" containerName="agent"
Jul 31 15:17:06.527832 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.527803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.530826 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.530804    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zszp\"/\"kube-root-ca.crt\""
Jul 31 15:17:06.531921 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.531902    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4zszp\"/\"default-dockercfg-7tg6t\""
Jul 31 15:17:06.532025 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.531905    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4zszp\"/\"openshift-service-ca.crt\""
Jul 31 15:17:06.535362 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.535333    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zszp/must-gather-b9kgh"]
Jul 31 15:17:06.538120 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.538089    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:17:06.567132 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.567087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gk5f\" (UniqueName: \"kubernetes.io/projected/296922b6-764e-4f0d-8f38-bba8bfe6654d-kube-api-access-4gk5f\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.567330 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.567150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/296922b6-764e-4f0d-8f38-bba8bfe6654d-must-gather-output\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.639427 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.639390    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:17:06.668707 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.668668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gk5f\" (UniqueName: \"kubernetes.io/projected/296922b6-764e-4f0d-8f38-bba8bfe6654d-kube-api-access-4gk5f\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.668876 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.668730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/296922b6-764e-4f0d-8f38-bba8bfe6654d-must-gather-output\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.669056 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.669040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/296922b6-764e-4f0d-8f38-bba8bfe6654d-must-gather-output\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.678520 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.678440    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gk5f\" (UniqueName: \"kubernetes.io/projected/296922b6-764e-4f0d-8f38-bba8bfe6654d-kube-api-access-4gk5f\") pod \"must-gather-b9kgh\" (UID: \"296922b6-764e-4f0d-8f38-bba8bfe6654d\") " pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.838283 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.838247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zszp/must-gather-b9kgh"
Jul 31 15:17:06.980922 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.980757    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zszp/must-gather-b9kgh"]
Jul 31 15:17:06.983686 ip-10-0-143-227 kubenswrapper[2575]: W0731 15:17:06.983654    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod296922b6_764e_4f0d_8f38_bba8bfe6654d.slice/crio-9163df2a1181b8358365c4f3298ddb354d9b5610f4b8452ed55e58ee19d1c534 WatchSource:0}: Error finding container 9163df2a1181b8358365c4f3298ddb354d9b5610f4b8452ed55e58ee19d1c534: Status 404 returned error can't find the container with id 9163df2a1181b8358365c4f3298ddb354d9b5610f4b8452ed55e58ee19d1c534
Jul 31 15:17:06.985695 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:06.985672    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:17:07.493024 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:07.492985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zszp/must-gather-b9kgh" event={"ID":"296922b6-764e-4f0d-8f38-bba8bfe6654d","Type":"ContainerStarted","Data":"9163df2a1181b8358365c4f3298ddb354d9b5610f4b8452ed55e58ee19d1c534"}
Jul 31 15:17:10.513648 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:10.513584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zszp/must-gather-b9kgh" event={"ID":"296922b6-764e-4f0d-8f38-bba8bfe6654d","Type":"ContainerStarted","Data":"7503b0674679b15b99dc5dab68723240f451c74fa86c6beb0eb1c6eb1bddbda7"}
Jul 31 15:17:10.514158 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:10.513667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zszp/must-gather-b9kgh" event={"ID":"296922b6-764e-4f0d-8f38-bba8bfe6654d","Type":"ContainerStarted","Data":"ece63648778710735aae73993d663f2a5e42112ffaf6cf082a534b17efafb51a"}
Jul 31 15:17:10.534149 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:10.534084    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4zszp/must-gather-b9kgh" podStartSLOduration=2.041640401 podStartE2EDuration="4.534066058s" totalImagesPullingTime="2.492425657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:10.532487437 +0000 UTC m=+1753.866607363" watchObservedRunningTime="2026-07-31 15:17:10.534066058 +0000 UTC m=+1753.868185981"
Jul 31 15:17:10.963533 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:10.963500    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nt9j7_94357ff7-eec1-4ce0-b983-f574a7204766/global-pull-secret-syncer/0.log"
Jul 31 15:17:11.067090 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:11.067051    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log"
Jul 31 15:17:11.213770 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:11.213676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-227.ec2.internal_caa64f70662f34d6016d67055768843e/haproxy/0.log"
Jul 31 15:17:12.204857 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:12.204817    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:17:12.264437 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:12.264406    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rgwjh_b77bb15f-e387-4e54-a709-c306d8e35922/konnectivity-agent/0.log" limit=1024
Jul 31 15:17:16.213127 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:16.213095    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7756dfdb96-l8lj8_b6a18a5d-30cb-408b-b700-f3586e60c275/metrics-server/0.log"
Jul 31 15:17:16.290856 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:16.290815    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cf5wf_875788cb-55ed-4a00-a657-4a418eb96d63/node-exporter/0.log"
Jul 31 15:17:16.351702 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:16.351649    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cf5wf_875788cb-55ed-4a00-a657-4a418eb96d63/kube-rbac-proxy/0.log"
Jul 31 15:17:16.425258 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:16.425232    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cf5wf_875788cb-55ed-4a00-a657-4a418eb96d63/init-textfile/0.log"
Jul 31 15:17:16.983864 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:16.983834    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-l9hxs_e8311c6e-336c-4254-89e7-e4bee8e6c877/prometheus-operator/0.log"
Jul 31 15:17:17.022910 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.022877    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-l9hxs_e8311c6e-336c-4254-89e7-e4bee8e6c877/kube-rbac-proxy/0.log"
Jul 31 15:17:17.080080 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.080052    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-srsgc_ce21b351-9a55-4290-92e2-d5836715aa40/prometheus-operator-admission-webhook/0.log"
Jul 31 15:17:17.239558 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.239467    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/thanos-query/0.log"
Jul 31 15:17:17.269000 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.268968    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/kube-rbac-proxy-web/0.log"
Jul 31 15:17:17.305257 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.305223    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/kube-rbac-proxy/0.log"
Jul 31 15:17:17.341560 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.341523    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/prom-label-proxy/0.log"
Jul 31 15:17:17.374447 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.374412    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/kube-rbac-proxy-rules/0.log"
Jul 31 15:17:17.406822 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:17.406792    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7fb9c6997f-vknzq_410bfc4b-b27a-4740-b7a9-ecfd14529b0b/kube-rbac-proxy-metrics/0.log"
Jul 31 15:17:19.542346 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.542295    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6f898d6645-h46kp_6a7f172c-249e-43d8-b95a-9bda31c36d40/console/0.log"
Jul 31 15:17:19.600505 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.600466    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"]
Jul 31 15:17:19.605681 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.605647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.615744 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.615708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"]
Jul 31 15:17:19.688950 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.688908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czz9k\" (UniqueName: \"kubernetes.io/projected/09a12c39-9266-4042-857b-ed83daa93c48-kube-api-access-czz9k\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.689168 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.688987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-sys\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.689168 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.689016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-lib-modules\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.689168 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.689113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-podres\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.689372 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.689223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-proc\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.790312 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.790265    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czz9k\" (UniqueName: \"kubernetes.io/projected/09a12c39-9266-4042-857b-ed83daa93c48-kube-api-access-czz9k\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.790627 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.790585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-sys\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.790755 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.790739    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-lib-modules\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.790885 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.790871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-podres\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.790994 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.790961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-sys\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.791177 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.791015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-proc\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.791343 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.791075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-proc\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.791471 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.791176    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-lib-modules\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.791471 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.791236    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09a12c39-9266-4042-857b-ed83daa93c48-podres\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.799796 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.799718    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czz9k\" (UniqueName: \"kubernetes.io/projected/09a12c39-9266-4042-857b-ed83daa93c48-kube-api-access-czz9k\") pod \"perf-node-gather-daemonset-f24jh\" (UID: \"09a12c39-9266-4042-857b-ed83daa93c48\") " pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:19.917308 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:19.917266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:20.105592 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:20.105555    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"]
Jul 31 15:17:20.580809 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:20.580688    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh" event={"ID":"09a12c39-9266-4042-857b-ed83daa93c48","Type":"ContainerStarted","Data":"d6102a09093b1fbcbf71860773a708795dea630a262e6f887b7b919354c16f9c"}
Jul 31 15:17:20.580809 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:20.580764    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh" event={"ID":"09a12c39-9266-4042-857b-ed83daa93c48","Type":"ContainerStarted","Data":"bff492c0090788258394546bc8201f0ccf02170ee6ecfbeed1d86d55c2f09c13"}
Jul 31 15:17:20.581613 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:20.581567    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:20.600171 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:20.600112    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh" podStartSLOduration=1.600095117 podStartE2EDuration="1.600095117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:20.598588291 +0000 UTC m=+1763.932708266" watchObservedRunningTime="2026-07-31 15:17:20.600095117 +0000 UTC m=+1763.934215066"
Jul 31 15:17:21.158683 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:21.158649    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czhw9_558d0e42-9039-431d-b66d-51770b2ef959/dns/0.log"
Jul 31 15:17:21.182912 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:21.182882    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-czhw9_558d0e42-9039-431d-b66d-51770b2ef959/kube-rbac-proxy/0.log"
Jul 31 15:17:21.261228 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:21.261196    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94z8k_d99871e8-7e77-436c-8d5b-56dbcac433b4/dns-node-resolver/0.log"
Jul 31 15:17:21.752305 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:21.752261    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-679dfb877-kg964_291bd47a-9d3f-461f-8a23-64c5cb07ebbd/registry/0.log"
Jul 31 15:17:21.801696 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:21.801662    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jpxk6_50c437ff-669d-4c9f-aa01-039af0c80380/node-ca/0.log"
Jul 31 15:17:22.969407 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:22.969376    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zx94w_22d806cf-8c2e-4a36-a22f-f7e32bb1f619/serve-healthcheck-canary/0.log"
Jul 31 15:17:23.449034 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:23.448995    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w2zm_ebf922ca-e856-43c9-8e88-899512ec8fbe/kube-rbac-proxy/0.log"
Jul 31 15:17:23.475384 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:23.475351    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w2zm_ebf922ca-e856-43c9-8e88-899512ec8fbe/exporter/0.log"
Jul 31 15:17:23.502985 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:23.502933    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2w2zm_ebf922ca-e856-43c9-8e88-899512ec8fbe/extractor/0.log"
Jul 31 15:17:26.047030 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:26.046991    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-swqdt_5c13c4ca-b802-4903-9bf0-25935c3fb9cf/manager/0.log"
Jul 31 15:17:26.299428 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:26.299336    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-s7pdd_91f69b60-00f0-46c9-ba94-3f7989788237/manager/0.log"
Jul 31 15:17:26.348284 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:26.348245    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-69l7z_1dd029e8-7192-454e-9abf-5e04ba1f8060/s3-init/0.log"
Jul 31 15:17:26.370747 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:26.370723    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-69l7z_1dd029e8-7192-454e-9abf-5e04ba1f8060/download-hf-model/0.log"
Jul 31 15:17:26.405951 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:26.405918    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6pgxz_f42d126c-ac98-4e13-8815-00a84765897b/seaweedfs/0.log"
Jul 31 15:17:27.602834 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:27.602804    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4zszp/perf-node-gather-daemonset-f24jh"
Jul 31 15:17:30.969410 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:30.969379    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d4jdn_3b076be2-0dd9-4f62-bef3-e00ecb467b25/migrator/0.log"
Jul 31 15:17:31.000714 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:31.000682    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-d4jdn_3b076be2-0dd9-4f62-bef3-e00ecb467b25/graceful-termination/0.log"
Jul 31 15:17:32.400492 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.400464    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:17:32.428680 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.428642    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/egress-router-binary-copy/0.log"
Jul 31 15:17:32.462318 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.462286    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/cni-plugins/0.log"
Jul 31 15:17:32.494488 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.494451    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/bond-cni-plugin/0.log"
Jul 31 15:17:32.521881 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.521846    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/routeoverride-cni/0.log"
Jul 31 15:17:32.551313 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.551281    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/whereabouts-cni-bincopy/0.log"
Jul 31 15:17:32.578789 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:32.578758    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5shsp_56403f34-c078-4c81-805c-7882ec9e5284/whereabouts-cni/0.log"
Jul 31 15:17:33.078524 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:33.078494    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-h72g8_380b53c0-18c5-44e1-b46d-3a0f59e5f975/kube-multus/0.log"
Jul 31 15:17:33.266510 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:33.266479    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xcfwb_61802f40-474e-4d7d-94ef-a3bfc53e6a6f/network-metrics-daemon/0.log"
Jul 31 15:17:33.292213 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:33.292180    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xcfwb_61802f40-474e-4d7d-94ef-a3bfc53e6a6f/kube-rbac-proxy/0.log"
Jul 31 15:17:34.388380 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.388347    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/ovn-controller/0.log"
Jul 31 15:17:34.406391 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.406365    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/ovn-acl-logging/0.log"
Jul 31 15:17:34.431378 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.431341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/kube-rbac-proxy-node/0.log"
Jul 31 15:17:34.455063 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.455030    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/kube-rbac-proxy-ovn-metrics/0.log"
Jul 31 15:17:34.477422 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.477385    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/northd/0.log"
Jul 31 15:17:34.500425 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.500395    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/nbdb/0.log"
Jul 31 15:17:34.523636 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.523591    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/sbdb/0.log"
Jul 31 15:17:34.585658 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:34.585620    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5w4ll_de489f2a-dbf3-4972-a8f1-84c98463eac5/ovnkube-controller/0.log"
Jul 31 15:17:35.829072 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:35.828992    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-65bdbbff45-d724s_a97a064f-de87-4f33-9387-5384c5967a2e/check-endpoints/0.log"
Jul 31 15:17:35.913460 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:35.913432    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dt4ln_0465a7ba-0b2d-4366-bbdf-27bacfa77a2c/network-check-target-container/0.log"
Jul 31 15:17:36.861279 ip-10-0-143-227 kubenswrapper[2575]: I0731 15:17:36.861196    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5psbp_a21ec7f6-eff2-4c88-8394-dca2df93ad25/iptables-alerter/0.log"