Jul 20 09:55:12.223761 ip-10-0-143-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 20 09:55:12.223773 ip-10-0-143-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 20 09:55:12.223780 ip-10-0-143-155 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 20 09:55:12.224012 ip-10-0-143-155 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 20 09:55:22.447736 ip-10-0-143-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 20 09:55:22.447752 ip-10-0-143-155 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot da80e56b80144844aabf998ad246f63f --
Jul 20 09:56:54.441500 ip-10-0-143-155 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 09:56:54.926561 ip-10-0-143-155 systemd[1]: Started Kubernetes Kubelet.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.841835    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847397    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847412    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847415    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847419    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847423    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847426    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847429    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:54.950035 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847432    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847434    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847437    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847439    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847442    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847445    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847447    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847450    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847452    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847455    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847457    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847460    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847463    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847466    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847469    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847471    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847474    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847477    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847479    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.209084 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847482    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847484    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847487    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847490    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847493    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847496    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847500    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847502    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847504    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847507    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847510    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847512    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847516    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847519    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847522    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847524    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847527    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847529    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847532    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847534    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847537    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.209842 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847539    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847542    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847545    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847548    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847551    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847554    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847556    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847559    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847561    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847564    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847566    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847569    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847571    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847574    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847579    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847583    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847586    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847589    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847592    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.210652 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847595    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847599    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847602    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847605    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847608    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847610    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847613    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847616    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847619    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847622    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847626    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847629    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847631    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847634    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847637    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847640    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847642    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847645    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847648    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847651    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.211326 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847654    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847656    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847659    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847661    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847664    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847667    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847669    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847672    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847675    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847677    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847680    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847683    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.847685    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848400    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848408    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848411    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848413    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848416    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848419    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848421    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.211942 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848424    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848426    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848429    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848431    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848434    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848437    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848439    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848442    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848444    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848450    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848453    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848457    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848462    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848465    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848468    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848471    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848474    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848476    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848479    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.212667 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848481    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848484    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848486    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848489    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848491    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848494    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848496    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848499    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848501    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848504    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848506    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848509    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848512    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848514    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848516    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848519    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848522    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848524    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848526    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848529    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.213552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848531    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848534    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848538    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848540    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848543    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848546    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848549    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848552    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848554    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848557    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848561    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848566    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848569    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848572    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848575    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848578    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848580    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848583    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848586    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848588    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.214179 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848591    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848593    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848596    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848598    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848601    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848603    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848606    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848608    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848611    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848613    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848616    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848618    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848621    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848623    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848628    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848632    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848635    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848638    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848641    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.214829 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848644    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848648    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848651    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848655    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848657    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848660    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848662    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848665    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848667    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848670    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848673    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848675    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848677    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.848680    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849049    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849054    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849056    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849059    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849062    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849064    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.215464 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849067    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849070    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849072    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849075    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849077    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849080    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849082    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849087    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849089    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849092    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849095    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849098    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849100    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849103    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849105    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849109    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849112    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849114    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849117    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849119    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:55.216094 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849121    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849124    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849127    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849129    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849132    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849134    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849137    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849139    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849142    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849144    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849147    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849149    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849151    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849154    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849157    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849159    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849161    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849164    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849166    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.216734 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849170    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849172    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849175    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849180    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849184    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849187    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849189    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849192    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849196    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849198    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849202    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849206    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849208    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849211    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849213    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849216    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849219    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849221    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849224    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.217376 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849226    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849229    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849247    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849250    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849252    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849255    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849257    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849259    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849262    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849264    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849267    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849269    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849272    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849276    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849278    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849282    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849285    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849287    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849291    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849294    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.217996 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849296    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849300    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849303    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849305    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849308    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849311    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849313    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849316    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849319    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849322    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849325    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849327    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849330    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849332    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.849335    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849408    2546 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849415    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849420    2546 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849425    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849429    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849432    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 09:56:55.218578 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849436    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849441    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849444    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849447    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849451    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849454    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849458    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849461    2546 flags.go:64] FLAG: --cgroup-root=""
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849464    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849467    2546 flags.go:64] FLAG: --client-ca-file=""
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849470    2546 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849473    2546 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849479    2546 flags.go:64] FLAG: --cluster-domain=""
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849482    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849485    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849488    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849491    2546 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849495    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849498    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849501    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849504    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849508    2546 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849511    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849513    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 09:56:55.219205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849516    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849520    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849524    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849527    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849530    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849532    2546 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849535    2546 flags.go:64] FLAG: --enable-server="true"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849538    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849542    2546 flags.go:64] FLAG: --event-burst="100"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849545    2546 flags.go:64] FLAG: --event-qps="50"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849548    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849551    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849556    2546 flags.go:64] FLAG: --eviction-hard=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849561    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849564    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849566    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849569    2546 flags.go:64] FLAG: --eviction-soft=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849572    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849575    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849578    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849584    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849587    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849590    2546 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849593    2546 flags.go:64] FLAG: --feature-gates=""
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849596    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 09:56:55.220060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849599    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849602    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849606    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849609    2546 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849612    2546 flags.go:64] FLAG: --help="false"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849615    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849618    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849621    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849624    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849627    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849631    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849633    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849636    2546 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849639    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849642    2546 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849645    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849648    2546 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849650    2546 flags.go:64] FLAG: --kube-reserved=""
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849653    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849657    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849660    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849663    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849666    2546 flags.go:64] FLAG: --lock-file=""
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849669    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 09:56:55.220944 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849672    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849675    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849680    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849684    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849687    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849690    2546 flags.go:64] FLAG: --logging-format="text"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849693    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849696    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849699    2546 flags.go:64] FLAG: --manifest-url=""
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849702    2546 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849706    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849709    2546 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849713    2546 flags.go:64] FLAG: --max-pods="110"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849716    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849719    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849722    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849725    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849728    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849731    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849733    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849740    2546 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849743    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849746    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 09:56:55.221834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849749    2546 flags.go:64] FLAG: --pod-cidr=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849752    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849758    2546 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849761    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849765    2546 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849769    2546 flags.go:64] FLAG: --port="10250"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849772    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849775    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2e30fc973e46efa"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849779    2546 flags.go:64] FLAG: --qos-reserved=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849781    2546 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849784    2546 flags.go:64] FLAG: --register-node="true"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849787    2546 flags.go:64] FLAG: --register-with-taints=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849792    2546 flags.go:64] FLAG: --registry-burst="10"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849795    2546 flags.go:64] FLAG: --registry-qps="5"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849798    2546 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849801    2546 flags.go:64] FLAG: --reserved-memory=""
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849804    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849807    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849810    2546 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849813    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849816    2546 flags.go:64] FLAG: --runonce="false"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849819    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849822    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849824    2546 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849827    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 09:56:55.222695 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849830    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849833    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849836    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849840    2546 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849842    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849845    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849848    2546 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849851    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849854    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849857    2546 flags.go:64] FLAG: --system-cgroups=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849860    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849865    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849869    2546 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849872    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849876    2546 flags.go:64] FLAG: --tls-min-version=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849879    2546 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849882    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849886    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849888    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849892    2546 flags.go:64] FLAG: --v="2"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849896    2546 flags.go:64] FLAG: --version="false"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849901    2546 flags.go:64] FLAG: --vmodule=""
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849904    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.849907    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850018    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.225852 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850022    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850026    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850029    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850032    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850036    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850040    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850044    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850048    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850052    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850056    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850058    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850061    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850064    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850066    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850069    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850071    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850074    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850077    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850079    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850083    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.226693 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850085    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850088    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850090    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850093    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850096    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850098    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850102    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850104    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850107    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850110    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850113    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850116    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850118    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850122    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850126    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850128    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850131    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850134    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850137    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850139    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.320332 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850142    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850144    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850147    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850149    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850151    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850154    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850156    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850159    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850161    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850164    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850167    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850171    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850173    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850176    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850178    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850180    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850183    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850185    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850189    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.321047 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850192    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850194    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850197    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850200    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850202    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850205    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850207    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850210    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850212    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850215    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850217    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850220    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850222    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850224    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850227    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850242    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850245    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850248    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850250    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850253    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.321796 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850256    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850259    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850261    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850265    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850270    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850273    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850276    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850279    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850281    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850284    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850286    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850290    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850294    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850297    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850301    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850304    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850307    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850310    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.322514 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.850313    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.850986    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.857644    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.857658    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.858457    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.858469    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858523    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858529    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858533    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858537    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858540    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858543    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858547    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858549    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858552    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.323250 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858555    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858557    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858560    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858562    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858565    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858568    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858573    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858576    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858578    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858581    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858583    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858586    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858589    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858591    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858594    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858597    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858599    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858603    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858606    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858609    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.323957 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858612    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858615    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858618    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858621    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858623    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858626    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858630    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858634    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858636    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858639    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858641    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858644    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858647    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858649    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858652    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858655    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858657    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858660    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858666    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858668    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.324677 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858671    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858673    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858678    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858681    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858684    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858686    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858690    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858693    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858696    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858700    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858704    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858708    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858711    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858715    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858717    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858720    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858722    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858725    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858727    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.325407 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858730    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858732    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858735    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858738    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858740    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858742    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858745    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858748    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858750    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858753    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858755    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858759    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858762    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858765    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858768    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858770    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858772    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858775    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858778    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858780    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.326115 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858783    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858785    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858789    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858792    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858795    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858797    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858800    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858803    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858805    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858808    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858811    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.858816    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858923    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858929    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858931    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 09:56:55.326848 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858934    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858936    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858939    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858942    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858945    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858949    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858952    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858955    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858959    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858962    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858965    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858968    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858970    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858973    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858976    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858979    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858981    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858984    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858987    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858990    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 09:56:55.327603 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858994    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858996    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.858999    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859001    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859004    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859007    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859010    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859012    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859015    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859017    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859020    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859026    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859029    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859031    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859034    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859036    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859039    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859041    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859044    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859047    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 09:56:55.328388 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859050    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859053    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859055    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859058    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859060    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859063    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859065    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859068    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859070    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859073    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859075    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859078    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859081    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859084    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859087    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859089    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859092    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859094    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859097    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859100    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 09:56:55.329168 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859103    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859105    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859109    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859111    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859113    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859116    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859119    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859121    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859124    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859126    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859128    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859131    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859134    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859137    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859140    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859142    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859145    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859147    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859150    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859152    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 09:56:55.329918 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859155    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859157    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859160    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859162    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859167    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859170    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859173    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859175    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859178    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859181    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859183    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859186    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859189    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859191    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859196    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 09:56:55.330704 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:54.859199    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.859204    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.859311    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.862398    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.863737    2546 server.go:1019] "Starting client certificate rotation"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.863836    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.863873    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.866995    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.871634    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.884184    2546 log.go:25] "Validated CRI v1 runtime API"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.886224    2546 log.go:25] "Validated CRI v1 image API"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.888094    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 09:56:55.331471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.891119    2546 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 955ffdbf-c836-4245-ace9-4cdc95c9166b:/dev/nvme0n1p3 f61eed44-5464-4cd2-a740-b4aa30bfe715:/dev/nvme0n1p4]
Jul 20 09:56:55.331844 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.891132    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 09:56:55.331844 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.892787    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.897269    2546 manager.go:216] Machine: {Timestamp:2026-07-20 09:56:54.894829305 +0000 UTC m=+0.333087938 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100576 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f302647d937c68cb36a6d4fb18877 SystemUUID:ec2f3026-47d9-37c6-8cb3-6a6d4fb18877 BootID:da80e56b-8014-4844-aabf-998ad246f63f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:c0:ca:4d:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:c0:ca:4d:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:8c:bb:75:f1:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.897376    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.897455    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.901310    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.901338    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.902620    2546 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.902631    2546 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.902637    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.903400    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.905272    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908538    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908647    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908681    2546 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908697    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908715    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908724    2546 kubelet.go:398] "Adding apiserver pod source"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.908732    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 09:56:55.332574 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.909046    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.909069    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.911636    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.913627    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.913655    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915646    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915660    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915666    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915677    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915689    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915695    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915703    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915709    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915716    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915721    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915733    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.915738    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.916400    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bqg22"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.917154    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.917162    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.918994    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.919588    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.919589    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.920044    2546 server.go:1268] "Started kubelet"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.920128    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.920113    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.924300    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.924861    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bqg22"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.926457    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.926460    2546 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.931790    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.931163    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-155.ec2.internal.18c3f6f64347e0e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-155.ec2.internal,UID:ip-10-0-143-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-155.ec2.internal,},FirstTimestamp:2026-07-20 09:56:54.920020196 +0000 UTC m=+0.358278829,LastTimestamp:2026-07-20 09:56:54.920020196 +0000 UTC m=+0.358278829,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-155.ec2.internal,}"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.934546    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.934553    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.934793    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935310    2546 factory.go:55] Registering systemd factory
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935324    2546 factory.go:223] Registration of the systemd container factory successfully
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935512    2546 factory.go:153] Registering CRI-O factory
Jul 20 09:56:55.333291 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935563    2546 factory.go:223] Registration of the crio container factory successfully
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935614    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935635    2546 factory.go:103] Registering Raw factory
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935650    2546 manager.go:1196] Started watching for new ooms in manager
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935929    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935933    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.935950    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.936062    2546 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.936069    2546 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.936091    2546 manager.go:349] Starting recovery of all containers
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.938728    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:54.949510    2546 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-155.ec2.internal\" not found" node="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.951530    2546 manager.go:354] Recovery completed
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.955451    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.960893    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.960920    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.960933    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.961487    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.961497    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.961499    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.961519    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.962701    2546 policy_none.go:49] "None policy: Start"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.962715    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.962724    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.963662    2546 policy_none.go:47] "Start"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.963670    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.963695    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:54.963704    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.004269    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.005885    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.006123    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.006371    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.006404    2546 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.008454    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.035034    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.043780    2546 manager.go:337] "Starting Device Plugin manager"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.043809    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.043821    2546 server.go:87] "Starting device plugin registration server"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.044061    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.044072    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.044160    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.044285    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.044295    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.044632    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.044666    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.107488    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"]
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.107560    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.108401    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.108425    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.337689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.108434    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.109678    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.109803    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.109833    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110300    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110319    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110334    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110383    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110401    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.110415    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.125705    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-155.ec2.internal\" not found" node="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.136278    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.136306    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.136330    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48c98dd19a1c52c34781facdd3c26946-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"48c98dd19a1c52c34781facdd3c26946\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.145098    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.145753    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.145775    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.145784    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.145804    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.151927    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.151945    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-155.ec2.internal\": node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.163957    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.172071    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.172095    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.174552    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.174576    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.174585    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeHasSufficientPID"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.183226    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-155.ec2.internal\" not found" node="ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.339127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236588    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236650    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48c98dd19a1c52c34781facdd3c26946-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"48c98dd19a1c52c34781facdd3c26946\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236673    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/053798e0d93501c34929affbf8ec0b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal\" (UID: \"053798e0d93501c34929affbf8ec0b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.236723    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48c98dd19a1c52c34781facdd3c26946-config\") pod \"kube-apiserver-proxy-ip-10-0-143-155.ec2.internal\" (UID: \"48c98dd19a1c52c34781facdd3c26946\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.264012    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.436598 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.364478    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.544864 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.465286    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.544864 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.475446    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.544864 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.486386    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:55.637509 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.565847    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.738505 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.666340    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.843227 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.766835    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.863352    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.863498    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.863524    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.867478    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.930270    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 09:51:56 +0000 UTC" deadline="2028-03-04 12:30:24.387451887 +0000 UTC"
Jul 20 09:56:55.942008 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:55.930297    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14234h33m28.457158663s"
Jul 20 09:56:56.046827 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:55.967797    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:56.145827 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:56.062629    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:56.145827 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:56.068081    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:56.248341 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:56.168512    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:56.268990    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-155.ec2.internal\" not found"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:57.721443    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:57.725917    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:57.737733    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:57.738676    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal"
Jul 20 09:56:57.751566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:57.746141    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 09:56:58.312025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.300901    2546 apiserver.go:52] "Watching apiserver"
Jul 20 09:56:58.313851 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.312250    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 09:56:58.313851 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.313430    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-cqlg8","kube-system/konnectivity-agent-4q64h","kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8q2vn","openshift-dns/node-resolver-4l429","openshift-image-registry/node-ca-22smx","openshift-multus/multus-additional-cni-plugins-ctm7j","openshift-multus/network-metrics-daemon-5qzr2","openshift-network-diagnostics/network-check-target-5d4qt","kube-system/global-pull-secret-syncer-q7gmb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal","openshift-multus/multus-27ggd","openshift-network-operator/iptables-alerter-rsgch"]
Jul 20 09:56:58.315885 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.314821    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.316473    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.317813    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.317883    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.317958    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.317885    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 09:56:58.318111 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.318055    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6xb22\""
Jul 20 09:56:58.318382 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.318310    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.318537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.318526    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.319575 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.319559    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.321331 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321199    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.321331 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321272    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-99d92\""
Jul 20 09:56:58.321331 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321299    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 09:56:58.321551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321535    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.321622 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321607    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.321747 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321718    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 09:56:58.321826 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.321806    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xd8hj\""
Jul 20 09:56:58.322357 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.322344    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.322775 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.322744    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ls47k\""
Jul 20 09:56:58.322775 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.322771    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.322898 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.322775    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.323570 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.323550    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.323662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.323577    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jc2l2\""
Jul 20 09:56:58.323720 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.323682    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:56:58.323786 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.323760    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:56:58.323852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.323799    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.324890 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.324874    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.325257 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.325240    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.325330 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.325260    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z7hcn\""
Jul 20 09:56:58.325330 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.325303    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 09:56:58.325480 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.325347    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.326004 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.325990    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.326146 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.326126    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 09:56:58.327577 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.327560    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 09:56:58.328358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.328086    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.328358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.328238    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 09:56:58.328358 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.328251    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:56:58.328537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.328464    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 09:56:58.329313 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.329293    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.329508 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.329492    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.329690 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.329677    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 09:56:58.329935 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.329919    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wjjfp\""
Jul 20 09:56:58.330847 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.330181    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 09:56:58.330847 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.330422    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rzlrp\""
Jul 20 09:56:58.330847 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.330559    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 09:56:58.330847 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.330798    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.331846 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.331830    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.331900 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.331886    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:56:58.333387 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.333369    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h84g7\""
Jul 20 09:56:58.333551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.333538    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 09:56:58.333717 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.333706    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 09:56:58.333828 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.333815    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 09:56:58.335817 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335803    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-dbus\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.335858 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335824    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-os-release\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.335858 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.335946 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335856    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.335946 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335870    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-run\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.335946 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335933    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2hn\" (UniqueName: \"kubernetes.io/projected/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-kube-api-access-qk2hn\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336042 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335954    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-agent-certs\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.336042 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-host\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.336042 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.335995    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-k8s-cni-cncf-io\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336042 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336024    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-systemd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336043    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336061    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wph\" (UniqueName: \"kubernetes.io/projected/2578aede-60eb-4f71-94ed-06e90d6cf997-kube-api-access-r6wph\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336091    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-conf-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336104    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-var-lib-kubelet\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336117    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-config\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336134    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxmb\" (UniqueName: \"kubernetes.io/projected/92cf6d64-b819-4546-8442-039af4c1b4c9-kube-api-access-fcxmb\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336164    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-system-cni-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336188    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-socket-dir-parent\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336206    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-hostroot\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336223    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-etc-kubernetes\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336244    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-tmp\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336262    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-konnectivity-ca\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336275    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df265ea5-1894-4478-8759-17110f267128-tmp-dir\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336287    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gts\" (UniqueName: \"kubernetes.io/projected/df265ea5-1894-4478-8759-17110f267128-kube-api-access-f2gts\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336304    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bds4b\" (UniqueName: \"kubernetes.io/projected/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-kube-api-access-bds4b\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336325    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-modprobe-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336345    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-lib-modules\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336359    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-netns\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-var-lib-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336421 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336411    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336435    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336450    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-conf\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336462    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-netns\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336487    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-kubelet\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336505    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-tuned\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336523    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-script-lib\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336570    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-cnibin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336594    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-multus-certs\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336611    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-systemd\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336637    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-sys\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336650    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-ovn\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336664    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-netd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-kubernetes\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336703    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-system-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.336856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336722    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv25r\" (UniqueName: \"kubernetes.io/projected/17948cd8-1721-4f40-8576-c21cda2c71c6-kube-api-access-vv25r\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336762    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336790    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw26\" (UniqueName: \"kubernetes.io/projected/72706163-7492-4705-9a19-452370fc456b-kube-api-access-9hw26\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17948cd8-1721-4f40-8576-c21cda2c71c6-iptables-alerter-script\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336829    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df265ea5-1894-4478-8759-17110f267128-hosts-file\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336865    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-env-overrides\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336894    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-kubelet-config\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336920    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336941    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-daemon-config\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336969    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-kubelet\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.336986    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-bin\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337003    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-etc-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337019    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337035    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-os-release\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337048    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-bin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337085    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:56:58.337327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337108    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-log-socket\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysconfig\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337160    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-cni-binary-copy\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337176    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17948cd8-1721-4f40-8576-c21cda2c71c6-host-slash\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337189    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-host\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337208    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-serviceca\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337227    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-slash\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337257    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-cnibin\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337277    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-multus\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337320    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-systemd-units\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-node-log\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.337810 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.337369    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72706163-7492-4705-9a19-452370fc456b-ovn-node-metrics-cert\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.340311 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.340296    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 09:56:58.357021 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.357006    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l8ft8"
Jul 20 09:56:58.366653 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.366634    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l8ft8"
Jul 20 09:56:58.427386 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.427369    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 09:56:58.438276 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438252    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-dbus\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-os-release\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438386    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-os-release\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438425    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438439    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-dbus\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438452    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-run\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438493    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-run\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438487    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2hn\" (UniqueName: \"kubernetes.io/projected/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-kube-api-access-qk2hn\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-agent-certs\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.438553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438546    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-host\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438568    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-k8s-cni-cncf-io\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438591    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-systemd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438612    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438640    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wph\" (UniqueName: \"kubernetes.io/projected/2578aede-60eb-4f71-94ed-06e90d6cf997-kube-api-access-r6wph\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438687    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-conf-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-var-lib-kubelet\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-config\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438765    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxmb\" (UniqueName: \"kubernetes.io/projected/92cf6d64-b819-4546-8442-039af4c1b4c9-kube-api-access-fcxmb\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438787    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-system-cni-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438826    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-socket-dir-parent\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438851    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-hostroot\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438874    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-etc-kubernetes\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.438914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438896    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-tmp\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438927    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-konnectivity-ca\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.438981    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df265ea5-1894-4478-8759-17110f267128-tmp-dir\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439008    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gts\" (UniqueName: \"kubernetes.io/projected/df265ea5-1894-4478-8759-17110f267128-kube-api-access-f2gts\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439096    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-systemd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439113    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-hostroot\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439152    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-etc-kubernetes\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439164    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439216    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-socket-dir-parent\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439244    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-k8s-cni-cncf-io\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439317    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439439    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-conf-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439500    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-system-cni-dir\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.439553 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439548    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-var-lib-kubelet\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439587    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-host\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439699    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bds4b\" (UniqueName: \"kubernetes.io/projected/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-kube-api-access-bds4b\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439728    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-modprobe-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439753    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-lib-modules\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439761    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df265ea5-1894-4478-8759-17110f267128-tmp-dir\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439776    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-netns\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439802    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-var-lib-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439882    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-sys-fs\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-lib-modules\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439953    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-netns\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440000    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-var-lib-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-conf\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440048    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440058    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 09:56:58.440261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.439908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysctl-conf\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-modprobe-d\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440287    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-netns\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440414    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-kubelet\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440450    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-netns\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-kubelet\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440454    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-tuned\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-script-lib\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440587    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-cni-binary-copy\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440632    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440686    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-registration-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440737    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-cnibin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-device-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440788    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-multus-certs\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-systemd\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440831    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-sys\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-ovn\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.441138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440938    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-netd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440961    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-kubernetes\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-system-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.440993    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-konnectivity-ca\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441016    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv25r\" (UniqueName: \"kubernetes.io/projected/17948cd8-1721-4f40-8576-c21cda2c71c6-kube-api-access-vv25r\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441084    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-systemd\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441088    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw26\" (UniqueName: \"kubernetes.io/projected/72706163-7492-4705-9a19-452370fc456b-kube-api-access-9hw26\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-socket-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441168    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-script-lib\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441260    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-sys\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441270    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17948cd8-1721-4f40-8576-c21cda2c71c6-iptables-alerter-script\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441304    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df265ea5-1894-4478-8759-17110f267128-hosts-file\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441321    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-ovn\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441332    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-env-overrides\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-netd\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441363    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-kubelet-config\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441411    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441272    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-ovnkube-config\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441438    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-system-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441463    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-run-multus-certs\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441486    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-daemon-config\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441338    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-cnibin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.441585    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441636    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-kubernetes\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441717    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4142bda-92f5-4056-a55e-7d4bd77cd619-kubelet-config\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441765    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-kubelet\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441802    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df265ea5-1894-4478-8759-17110f267128-hosts-file\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.441961    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-bin\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.442773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-etc-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.442854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.442903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.442890    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-os-release\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.442920    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72706163-7492-4705-9a19-452370fc456b-env-overrides\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.442978    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-kubelet\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443015    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443039    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2578aede-60eb-4f71-94ed-06e90d6cf997-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443115    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-etc-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.443168    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:58.943124022 +0000 UTC m=+2.990548765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443864    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/17948cd8-1721-4f40-8576-c21cda2c71c6-iptables-alerter-script\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443489    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-cni-bin\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443685    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-os-release\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443936    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-bin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443982    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443991    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-tmp\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444014    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-log-socket\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443996    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-bin\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444057    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2mr\" (UniqueName: \"kubernetes.io/projected/4b095d3a-3625-4892-8729-c8e57740fe97-kube-api-access-mz2mr\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.444162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.443323    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-run-openvswitch\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444136    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2grmh\" (UniqueName: \"kubernetes.io/projected/44618c9c-a6c5-4e94-aedf-cadacba06315-kube-api-access-2grmh\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.444995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444170    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-log-socket\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.444995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444252    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-daemon-config\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.444995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.444367    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a-agent-certs\") pod \"konnectivity-agent-4q64h\" (UID: \"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a\") " pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.445908 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.445869    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysconfig\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.446030 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.445909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-tuned\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.446030 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.445932    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-cni-binary-copy\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446030 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446002    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-etc-sysconfig\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.446030 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446021    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17948cd8-1721-4f40-8576-c21cda2c71c6-host-slash\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446059    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-host\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446121    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-serviceca\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446148    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-slash\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446170    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-host\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446179    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-cnibin\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.446225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446207    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2578aede-60eb-4f71-94ed-06e90d6cf997-cnibin\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446256    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-host-slash\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446279    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-multus\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446299    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17948cd8-1721-4f40-8576-c21cda2c71c6-host-slash\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446336    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-host-var-lib-cni-multus\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446364    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92cf6d64-b819-4546-8442-039af4c1b4c9-multus-cni-dir\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446415    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-systemd-units\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446535 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92cf6d64-b819-4546-8442-039af4c1b4c9-cni-binary-copy\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.446843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-node-log\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446522    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-systemd-units\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72706163-7492-4705-9a19-452370fc456b-ovn-node-metrics-cert\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.446843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.446668    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72706163-7492-4705-9a19-452370fc456b-node-log\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.447098 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.447077    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-serviceca\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.449235 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.449207    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72706163-7492-4705-9a19-452370fc456b-ovn-node-metrics-cert\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.450029 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.449757    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gts\" (UniqueName: \"kubernetes.io/projected/df265ea5-1894-4478-8759-17110f267128-kube-api-access-f2gts\") pod \"node-resolver-4l429\" (UID: \"df265ea5-1894-4478-8759-17110f267128\") " pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.450029 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.449989    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxmb\" (UniqueName: \"kubernetes.io/projected/92cf6d64-b819-4546-8442-039af4c1b4c9-kube-api-access-fcxmb\") pod \"multus-27ggd\" (UID: \"92cf6d64-b819-4546-8442-039af4c1b4c9\") " pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.450329 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.450311    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw26\" (UniqueName: \"kubernetes.io/projected/72706163-7492-4705-9a19-452370fc456b-kube-api-access-9hw26\") pod \"ovnkube-node-cqlg8\" (UID: \"72706163-7492-4705-9a19-452370fc456b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.450406 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.450344    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wph\" (UniqueName: \"kubernetes.io/projected/2578aede-60eb-4f71-94ed-06e90d6cf997-kube-api-access-r6wph\") pod \"multus-additional-cni-plugins-ctm7j\" (UID: \"2578aede-60eb-4f71-94ed-06e90d6cf997\") " pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.450486 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.450467    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2hn\" (UniqueName: \"kubernetes.io/projected/6a62ec41-3d0d-462d-a0e8-fd55f5b3b565-kube-api-access-qk2hn\") pod \"tuned-8q2vn\" (UID: \"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565\") " pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.450749 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.450722    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bds4b\" (UniqueName: \"kubernetes.io/projected/742d5ccd-da6c-4f25-b376-9d9b85e9cee1-kube-api-access-bds4b\") pod \"node-ca-22smx\" (UID: \"742d5ccd-da6c-4f25-b376-9d9b85e9cee1\") " pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.451360 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.451334    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:58.451659 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.451630    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv25r\" (UniqueName: \"kubernetes.io/projected/17948cd8-1721-4f40-8576-c21cda2c71c6-kube-api-access-vv25r\") pod \"iptables-alerter-rsgch\" (UID: \"17948cd8-1721-4f40-8576-c21cda2c71c6\") " pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.453671 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.453649    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:58.453671 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.453665    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:58.453770 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.453674    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:58.453770 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.453713    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:56:58.953701201 +0000 UTC m=+3.001125927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:58.474002 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.473987    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-22smx"
Jul 20 09:56:58.488246 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.488229    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:56:58.506747 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.506717    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:56:58.547235 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547203    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.547373 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547351    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-sys-fs\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547444 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.547357    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:58.547444 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.547429    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:59.047409818 +0000 UTC m=+3.094834545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:58.547444 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547428    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-sys-fs\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-registration-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547491    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-device-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547541    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547558    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-registration-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547578    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-socket-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-device-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547627    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547667    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547679    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-etc-selinux\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547734    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2mr\" (UniqueName: \"kubernetes.io/projected/4b095d3a-3625-4892-8729-c8e57740fe97-kube-api-access-mz2mr\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547759    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4b095d3a-3625-4892-8729-c8e57740fe97-socket-dir\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.547852 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.547761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2grmh\" (UniqueName: \"kubernetes.io/projected/44618c9c-a6c5-4e94-aedf-cadacba06315-kube-api-access-2grmh\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.556284 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.556266    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2mr\" (UniqueName: \"kubernetes.io/projected/4b095d3a-3625-4892-8729-c8e57740fe97-kube-api-access-mz2mr\") pod \"aws-ebs-csi-driver-node-x6z4s\" (UID: \"4b095d3a-3625-4892-8729-c8e57740fe97\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.556395 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.556373    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2grmh\" (UniqueName: \"kubernetes.io/projected/44618c9c-a6c5-4e94-aedf-cadacba06315-kube-api-access-2grmh\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:58.584833 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.584801    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053798e0d93501c34929affbf8ec0b82.slice/crio-2c33e3af4b36629322c9224ffa5a04ac0993fb4cba05a2ab119d1f683eee1a34 WatchSource:0}: Error finding container 2c33e3af4b36629322c9224ffa5a04ac0993fb4cba05a2ab119d1f683eee1a34: Status 404 returned error can't find the container with id 2c33e3af4b36629322c9224ffa5a04ac0993fb4cba05a2ab119d1f683eee1a34
Jul 20 09:56:58.585302 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.585266    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c98dd19a1c52c34781facdd3c26946.slice/crio-3e8f559be56ddadc2d1b41bc8365a5f1330168889bf3327a95ce963abbc9f294 WatchSource:0}: Error finding container 3e8f559be56ddadc2d1b41bc8365a5f1330168889bf3327a95ce963abbc9f294: Status 404 returned error can't find the container with id 3e8f559be56ddadc2d1b41bc8365a5f1330168889bf3327a95ce963abbc9f294
Jul 20 09:56:58.590494 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.590477    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 09:56:58.675992 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.675972    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ctm7j"
Jul 20 09:56:58.684859 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.684836    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2578aede_60eb_4f71_94ed_06e90d6cf997.slice/crio-1c34dc997740bf5ab0d5f17ba0f59b58f7f6f0ba553fb4987e865302b98e9867 WatchSource:0}: Error finding container 1c34dc997740bf5ab0d5f17ba0f59b58f7f6f0ba553fb4987e865302b98e9867: Status 404 returned error can't find the container with id 1c34dc997740bf5ab0d5f17ba0f59b58f7f6f0ba553fb4987e865302b98e9867
Jul 20 09:56:58.694751 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.694727    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-27ggd"
Jul 20 09:56:58.701488 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.701468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsgch"
Jul 20 09:56:58.702314 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.702293    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cf6d64_b819_4546_8442_039af4c1b4c9.slice/crio-268db1df2266e6c29279a013f125dc88c50d87189c15805c37a4f90b5bcb28df WatchSource:0}: Error finding container 268db1df2266e6c29279a013f125dc88c50d87189c15805c37a4f90b5bcb28df: Status 404 returned error can't find the container with id 268db1df2266e6c29279a013f125dc88c50d87189c15805c37a4f90b5bcb28df
Jul 20 09:56:58.709767 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.709749    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17948cd8_1721_4f40_8576_c21cda2c71c6.slice/crio-2da63f944c0c8a265d059886963e4b0d6cf832aa0c770fd8e4b8d683b4f32a8f WatchSource:0}: Error finding container 2da63f944c0c8a265d059886963e4b0d6cf832aa0c770fd8e4b8d683b4f32a8f: Status 404 returned error can't find the container with id 2da63f944c0c8a265d059886963e4b0d6cf832aa0c770fd8e4b8d683b4f32a8f
Jul 20 09:56:58.732482 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.732438    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8q2vn"
Jul 20 09:56:58.737192 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.737177    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4l429"
Jul 20 09:56:58.739522 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.739504    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a62ec41_3d0d_462d_a0e8_fd55f5b3b565.slice/crio-49d826a011827f1ca1c9c0ec0dced88109739cbe06a3ab003a2befd47ea6ed7b WatchSource:0}: Error finding container 49d826a011827f1ca1c9c0ec0dced88109739cbe06a3ab003a2befd47ea6ed7b: Status 404 returned error can't find the container with id 49d826a011827f1ca1c9c0ec0dced88109739cbe06a3ab003a2befd47ea6ed7b
Jul 20 09:56:58.745887 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.745868    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf265ea5_1894_4478_8759_17110f267128.slice/crio-74b41769a2f78338e586af76973769be6213eae91e26f8095c4d1ce1a98d94d9 WatchSource:0}: Error finding container 74b41769a2f78338e586af76973769be6213eae91e26f8095c4d1ce1a98d94d9: Status 404 returned error can't find the container with id 74b41769a2f78338e586af76973769be6213eae91e26f8095c4d1ce1a98d94d9
Jul 20 09:56:58.792373 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.792354    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742d5ccd_da6c_4f25_b376_9d9b85e9cee1.slice/crio-fecc1815bdaf6ccd246993d06d35682971fdc02587c4cbdd3c17501dc809fb2a WatchSource:0}: Error finding container fecc1815bdaf6ccd246993d06d35682971fdc02587c4cbdd3c17501dc809fb2a: Status 404 returned error can't find the container with id fecc1815bdaf6ccd246993d06d35682971fdc02587c4cbdd3c17501dc809fb2a
Jul 20 09:56:58.812372 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.812351    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s"
Jul 20 09:56:58.820768 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.820750    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b095d3a_3625_4892_8729_c8e57740fe97.slice/crio-633a080cc979229288e63eff3667d228e53eae5c5d1e681bcfbdc749eaf15140 WatchSource:0}: Error finding container 633a080cc979229288e63eff3667d228e53eae5c5d1e681bcfbdc749eaf15140: Status 404 returned error can't find the container with id 633a080cc979229288e63eff3667d228e53eae5c5d1e681bcfbdc749eaf15140
Jul 20 09:56:58.842919 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.842888    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 09:56:58.851251 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.851232    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72706163_7492_4705_9a19_452370fc456b.slice/crio-ef359d481a51768eaad4a7def1bc4e9482b31a22a3b5a92df1fe47a1bcb13892 WatchSource:0}: Error finding container ef359d481a51768eaad4a7def1bc4e9482b31a22a3b5a92df1fe47a1bcb13892: Status 404 returned error can't find the container with id ef359d481a51768eaad4a7def1bc4e9482b31a22a3b5a92df1fe47a1bcb13892
Jul 20 09:56:58.935977 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:56:58.935956    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab90c7c_db4f_4f60_9bc5_fe643ce16a9a.slice/crio-53b350b67608234fc49954eff58a3ac316d4ddfc008e29afe2889d0c641bc916 WatchSource:0}: Error finding container 53b350b67608234fc49954eff58a3ac316d4ddfc008e29afe2889d0c641bc916: Status 404 returned error can't find the container with id 53b350b67608234fc49954eff58a3ac316d4ddfc008e29afe2889d0c641bc916
Jul 20 09:56:58.951085 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:58.951055    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:58.951224 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.951172    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:58.951286 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:58.951239    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:56:59.951218844 +0000 UTC m=+3.998643568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:59.023392 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.023294    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"ef359d481a51768eaad4a7def1bc4e9482b31a22a3b5a92df1fe47a1bcb13892"}
Jul 20 09:56:59.025316 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.025286    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4l429" event={"ID":"df265ea5-1894-4478-8759-17110f267128","Type":"ContainerStarted","Data":"74b41769a2f78338e586af76973769be6213eae91e26f8095c4d1ce1a98d94d9"}
Jul 20 09:56:59.026611 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.026592    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerStarted","Data":"1c34dc997740bf5ab0d5f17ba0f59b58f7f6f0ba553fb4987e865302b98e9867"}
Jul 20 09:56:59.027777 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.027746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"053798e0d93501c34929affbf8ec0b82","Type":"ContainerStarted","Data":"2c33e3af4b36629322c9224ffa5a04ac0993fb4cba05a2ab119d1f683eee1a34"}
Jul 20 09:56:59.029018 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.028993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4q64h" event={"ID":"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a","Type":"ContainerStarted","Data":"53b350b67608234fc49954eff58a3ac316d4ddfc008e29afe2889d0c641bc916"}
Jul 20 09:56:59.030121 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.030096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s" event={"ID":"4b095d3a-3625-4892-8729-c8e57740fe97","Type":"ContainerStarted","Data":"633a080cc979229288e63eff3667d228e53eae5c5d1e681bcfbdc749eaf15140"}
Jul 20 09:56:59.031709 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.031688    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22smx" event={"ID":"742d5ccd-da6c-4f25-b376-9d9b85e9cee1","Type":"ContainerStarted","Data":"fecc1815bdaf6ccd246993d06d35682971fdc02587c4cbdd3c17501dc809fb2a"}
Jul 20 09:56:59.033009 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.032984    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8q2vn" event={"ID":"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565","Type":"ContainerStarted","Data":"49d826a011827f1ca1c9c0ec0dced88109739cbe06a3ab003a2befd47ea6ed7b"}
Jul 20 09:56:59.034355 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.034297    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsgch" event={"ID":"17948cd8-1721-4f40-8576-c21cda2c71c6","Type":"ContainerStarted","Data":"2da63f944c0c8a265d059886963e4b0d6cf832aa0c770fd8e4b8d683b4f32a8f"}
Jul 20 09:56:59.035804 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.035781    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-27ggd" event={"ID":"92cf6d64-b819-4546-8442-039af4c1b4c9","Type":"ContainerStarted","Data":"268db1df2266e6c29279a013f125dc88c50d87189c15805c37a4f90b5bcb28df"}
Jul 20 09:56:59.037331 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.037310    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" event={"ID":"48c98dd19a1c52c34781facdd3c26946","Type":"ContainerStarted","Data":"3e8f559be56ddadc2d1b41bc8365a5f1330168889bf3327a95ce963abbc9f294"}
Jul 20 09:56:59.051938 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.051910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:56:59.052090 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.052060    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:56:59.052168 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052090    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:56:59.052168 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052113    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:56:59.052168 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052126    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:59.052317 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052180    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:57:00.052159551 +0000 UTC m=+4.099584278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:56:59.052317 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052251    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:59.052434 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.052332    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:00.052292938 +0000 UTC m=+4.099717664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:56:59.368512 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.368320    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:58 +0000 UTC" deadline="2028-01-29 19:46:02.760387879 +0000 UTC"
Jul 20 09:56:59.368512 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.368374    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13401h49m3.392017633s"
Jul 20 09:56:59.960795 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:56:59.960761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:56:59.961613 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.960912    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:56:59.961613 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:56:59.960973    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:01.960954465 +0000 UTC m=+6.008379196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:00.062031 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.061997    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:00.062219 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.062057    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:00.062219 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062193    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:00.062324 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062250    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.062232133 +0000 UTC m=+6.109656856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:00.062702 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062684    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:00.062702 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062707    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:00.062878 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062720    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:00.062878 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.062766    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:57:02.062753272 +0000 UTC m=+6.110178009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:00.368567 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.368483    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 09:51:58 +0000 UTC" deadline="2028-02-01 19:29:01.169559391 +0000 UTC"
Jul 20 09:57:00.368567 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.368526    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13473h32m0.801038121s"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.400031    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.400177    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.400368    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:00.400031    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.400486    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:00.400534 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:00.400532    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:01.981195 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:01.981125    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:01.981728 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:01.981275    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:01.981728 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:01.981345    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:05.981325942 +0000 UTC m=+10.028750673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:02.081834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:02.081790    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:02.082034 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:02.081852    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:02.082034 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.081961    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.082034 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.082019    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.081999685 +0000 UTC m=+10.129424411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:02.082884 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.082853    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:02.082884 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.082885    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:02.083044 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.082900    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.083044 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.082959    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:57:06.082943041 +0000 UTC m=+10.130367770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:02.400244 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:02.400161    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:02.400448 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.400298    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:02.400448 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:02.400366    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:02.400448 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.400431    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:02.400603 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:02.400479    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:02.400603 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:02.400542    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:04.398229    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:04.398241    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:04.398366    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:04.398433    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:04.398520    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:04.398646 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:04.398613    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:06.011558 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.011512    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:06.012027 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.011686    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.012027 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.011752    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:14.011732115 +0000 UTC m=+18.059156844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.112488    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.112553    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112689    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112732    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112748    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112753    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:14.112733617 +0000 UTC m=+18.160158346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112760    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.112838 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.112794    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:57:14.112783307 +0000 UTC m=+18.160208043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:06.399268 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.398971    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:06.399268 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.399011    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:06.399268 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:06.399115    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:06.399625 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.399583    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:06.399717 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.399655    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:06.399830 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:06.399788    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:08.400962 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:08.400885    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:08.401415 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:08.400887    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:08.401415 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:08.400993    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:08.401415 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:08.401009    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:08.401415 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:08.401084    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:08.401415 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:08.401168    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:10.397489 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:10.397452    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:10.397937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:10.397458    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:10.397937 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:10.397590    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:10.397937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:10.397621    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:10.397937 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:10.397713    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:10.397937 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:10.397786    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:12.397577 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:12.397542    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:12.397577 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:12.397578    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:12.398057 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:12.397548    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:12.398057 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:12.397678    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:12.398057 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:12.397737    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:12.398057 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:12.397821    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:14.076482 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.076452    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:14.076821 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.076546    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:14.076821 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.076591    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.076578272 +0000 UTC m=+34.124002994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:14.176889 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.176812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:14.176889 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.176873    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.176971    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.176991    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.177011    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.177021    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.177041    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.177018741 +0000 UTC m=+34.224443466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:14.177129 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.177063    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:57:30.177047761 +0000 UTC m=+34.224472488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:14.397756 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.397726    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:14.397941 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.397726    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:14.397941 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:14.397876    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:14.398057 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.397841    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:14.398057 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.397964    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:14.398183 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:14.398087    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:16.398033 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:16.398003    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:16.398504 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:16.398141    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:16.398504 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:16.398146    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:16.398504 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:16.398199    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:16.398651 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:16.398456    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:16.398651 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:16.398543    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:18.398009 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:18.397973    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:18.398009 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:18.397991    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:18.398487 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:18.397983    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:18.398487 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:18.398103    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:18.398487 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:18.398152    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:18.398487 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:18.398246    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:19.113023 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.112800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8q2vn" event={"ID":"6a62ec41-3d0d-462d-a0e8-fd55f5b3b565","Type":"ContainerStarted","Data":"3c7519227728bdb74c4869d62c9466fa42e9a3f4b8c502de13617fa395d63fa5"}
Jul 20 09:57:19.117615 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.117490    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-27ggd" event={"ID":"92cf6d64-b819-4546-8442-039af4c1b4c9","Type":"ContainerStarted","Data":"b2da18cbd2b319c875a73dbb1752e85a0255b2b05781326e850b10b18992fa00"}
Jul 20 09:57:19.122685 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.122658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" event={"ID":"48c98dd19a1c52c34781facdd3c26946","Type":"ContainerStarted","Data":"097126f2938ceb1cd5e2a76d29d849bbd126c1d2f519221d331b4f0e319b19be"}
Jul 20 09:57:19.131899 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.131872    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"f70ad7a9efd71036d6f0ffcfb70372dd9a328a0464f918118719ec73d98055e6"}
Jul 20 09:57:19.132009 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.131927    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8q2vn" podStartSLOduration=3.143759091 podStartE2EDuration="23.131911311s" totalImagesPullingTime="19.98815222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:19.131610405 +0000 UTC m=+23.179035151" watchObservedRunningTime="2026-07-20 09:57:19.131911311 +0000 UTC m=+23.179336057"
Jul 20 09:57:19.149811 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:19.149765    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-27ggd" podStartSLOduration=3.120319768 podStartE2EDuration="23.149751855s" totalImagesPullingTime="20.029432087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:19.149552614 +0000 UTC m=+23.196977360" watchObservedRunningTime="2026-07-20 09:57:19.149751855 +0000 UTC m=+23.197176600"
Jul 20 09:57:20.135947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.135671    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4q64h" event={"ID":"1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a","Type":"ContainerStarted","Data":"8bacb2f0619e4d0f6897cc7bb2ca926c5853c080de3bf4eefb872d2eb18e7d53"}
Jul 20 09:57:20.137226 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.137199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s" event={"ID":"4b095d3a-3625-4892-8729-c8e57740fe97","Type":"ContainerStarted","Data":"d3aa007bbd9c51ae397964fa7e5252fbbe7073f930393f3d7850830f75b7d38a"}
Jul 20 09:57:20.138688 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.138659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-22smx" event={"ID":"742d5ccd-da6c-4f25-b376-9d9b85e9cee1","Type":"ContainerStarted","Data":"cf80a3f08bca18614b5ff74e4a541efac91955dd06984d57bc2f41cc18c61877"}
Jul 20 09:57:20.140232 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.140185    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsgch" event={"ID":"17948cd8-1721-4f40-8576-c21cda2c71c6","Type":"ContainerStarted","Data":"7e37b883234b4326df3317c91e9fa4b16c6249f92440ce0d3f5d099678e0f863"}
Jul 20 09:57:20.143412 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.143390    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"15de37cdf44b713dc8f5115f7b2a5646ddc3c85c034b63aca23104cd99a51e88"}
Jul 20 09:57:20.143525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.143461    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"133a45925ace94b5c633bd68d046fb728b435034c5dc826f39c5a7461af0ea49"}
Jul 20 09:57:20.143525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.143512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"97f2172948bfec7e3266947c8396ef18770f6b0939d5ffc7752d10c636b0328d"}
Jul 20 09:57:20.143635 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.143560    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"e63e057ad9f8c1419e13638cd657a537fe561e1604ac55bee25d0d0168fda73b"}
Jul 20 09:57:20.143635 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.143601    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"c4318185c286d049646e1e8d0649ba556b9b286bf068050d4ca8b098790095fa"}
Jul 20 09:57:20.144769 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.144746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4l429" event={"ID":"df265ea5-1894-4478-8759-17110f267128","Type":"ContainerStarted","Data":"ae07e242261eea769c31d68f5bfa78f403a6ec691512042d69d1c8b08760e8d2"}
Jul 20 09:57:20.146432 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.146409    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="16bd0824bc7b67546c7c63c58708d2a924e4af6f0858d98dd9d8a810a7419118" exitCode=0
Jul 20 09:57:20.146521 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.146444    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"16bd0824bc7b67546c7c63c58708d2a924e4af6f0858d98dd9d8a810a7419118"}
Jul 20 09:57:20.147970 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.147950    2546 generic.go:356] "Generic (PLEG): container finished" podID="053798e0d93501c34929affbf8ec0b82" containerID="81b7bac238917f0ba7bd0d6c4ba60c5db2668e5e0dd83016d045858a3be4e06c" exitCode=0
Jul 20 09:57:20.148173 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.148152    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"053798e0d93501c34929affbf8ec0b82","Type":"ContainerDied","Data":"81b7bac238917f0ba7bd0d6c4ba60c5db2668e5e0dd83016d045858a3be4e06c"}
Jul 20 09:57:20.153561 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.153528    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4q64h" podStartSLOduration=4.3828663 podStartE2EDuration="24.153513018s" totalImagesPullingTime="19.770646718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:20.152557023 +0000 UTC m=+24.199981773" watchObservedRunningTime="2026-07-20 09:57:20.153513018 +0000 UTC m=+24.200937765"
Jul 20 09:57:20.153652 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.153596    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-155.ec2.internal" podStartSLOduration=23.153591437 podStartE2EDuration="23.153591437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:19.16515088 +0000 UTC m=+23.212575625" watchObservedRunningTime="2026-07-20 09:57:20.153591437 +0000 UTC m=+24.201016184"
Jul 20 09:57:20.195349 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.195163    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-22smx" podStartSLOduration=4.28083929 podStartE2EDuration="24.1951498s" totalImagesPullingTime="19.91431051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:20.168128841 +0000 UTC m=+24.215553585" watchObservedRunningTime="2026-07-20 09:57:20.1951498 +0000 UTC m=+24.242574536"
Jul 20 09:57:20.211288 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.211249    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rsgch" podStartSLOduration=4.21443698 podStartE2EDuration="24.211237517s" totalImagesPullingTime="19.996800537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:20.211103919 +0000 UTC m=+24.258528665" watchObservedRunningTime="2026-07-20 09:57:20.211237517 +0000 UTC m=+24.258662261"
Jul 20 09:57:20.227140 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.227106    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4l429" podStartSLOduration=4.24217328 podStartE2EDuration="24.227094581s" totalImagesPullingTime="19.984921301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:20.227027827 +0000 UTC m=+24.274452573" watchObservedRunningTime="2026-07-20 09:57:20.227094581 +0000 UTC m=+24.274519325"
Jul 20 09:57:20.397750 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.397651    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:20.397750 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.397651    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:20.398015 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.397773    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:20.398015 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:20.397767    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:20.398015 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:20.397849    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:20.398015 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:20.397922    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:20.807651 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:20.807631    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 09:57:21.151791 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.151708    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" event={"ID":"053798e0d93501c34929affbf8ec0b82","Type":"ContainerStarted","Data":"4a1a8d74bdf22c489add7248b8d252d6368d59ecb4051e473ed55ed3c365a2c4"}
Jul 20 09:57:21.153811 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.153787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s" event={"ID":"4b095d3a-3625-4892-8729-c8e57740fe97","Type":"ContainerStarted","Data":"13fa94b231295b8fb0716359115ea1a111d5b6e75df6e2050a116d2f2dd98a6f"}
Jul 20 09:57:21.168843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.168801    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-155.ec2.internal" podStartSLOduration=24.168788275 podStartE2EDuration="24.168788275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:21.168476612 +0000 UTC m=+25.215901356" watchObservedRunningTime="2026-07-20 09:57:21.168788275 +0000 UTC m=+25.216213019"
Jul 20 09:57:21.448229 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.448062    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T09:57:20.807648039Z","UUID":"3a077fd6-9778-4624-89ed-0a068b28cc0b","Handler":null,"Name":"","Endpoint":""}
Jul 20 09:57:21.449968 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.449936    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 09:57:21.450153 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:21.449980    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 09:57:22.159045 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:22.159007    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s" event={"ID":"4b095d3a-3625-4892-8729-c8e57740fe97","Type":"ContainerStarted","Data":"52130d8318901a72d5f2e82fc27449d2377dd455571e751e23c2fdd70728c215"}
Jul 20 09:57:22.163150 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:22.163127    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"ff13d8528fe3327bb855c0b017def5e4debc2d21ea1b5631527213f3c7c7e241"}
Jul 20 09:57:22.397966 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:22.397892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:22.397966 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:22.397933    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:22.398257 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:22.398012    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:22.398257 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:22.397892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:22.398257 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:22.398164    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:22.398257 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:22.398233    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:23.186251 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:23.186206    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x6z4s" podStartSLOduration=4.907472231 podStartE2EDuration="27.186190338s" totalImagesPullingTime="22.278718107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:23.184448538 +0000 UTC m=+27.231873284" watchObservedRunningTime="2026-07-20 09:57:23.186190338 +0000 UTC m=+27.233615083"
Jul 20 09:57:23.700856 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:23.700828    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:57:24.051753 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.051677    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:57:24.052252 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.052226    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:57:24.168166 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.168138    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4q64h"
Jul 20 09:57:24.397455 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.397389    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:24.397859 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.397389    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:24.397859 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:24.397486    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:24.397859 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:24.397587    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:24.397859 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:24.397389    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:24.397859 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:24.397704    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:25.172863 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.172682    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" event={"ID":"72706163-7492-4705-9a19-452370fc456b","Type":"ContainerStarted","Data":"537f427cd509f018487a240ef1fc8ce9012f589c141da7a6f2cce5ce34a90576"}
Jul 20 09:57:25.173043 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.172985    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:25.173845 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.173816    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:25.177307 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.177284    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="69874dd8a327b82006d05f828eda554bd6c5e7d4fc01d5cca30d5ecb6fbcfcf9" exitCode=0
Jul 20 09:57:25.177416 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.177362    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"69874dd8a327b82006d05f828eda554bd6c5e7d4fc01d5cca30d5ecb6fbcfcf9"}
Jul 20 09:57:25.189501 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.189485    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:25.206007 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:25.205971    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8" podStartSLOduration=9.035215359 podStartE2EDuration="29.205960163s" totalImagesPullingTime="20.170744804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:25.205687236 +0000 UTC m=+29.253111983" watchObservedRunningTime="2026-07-20 09:57:25.205960163 +0000 UTC m=+29.253384907"
Jul 20 09:57:26.179482 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.179445    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:26.193795 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.193774    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:26.398442 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.398418    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:26.398563 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.398525    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:26.398563 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:26.398528    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:26.398563 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.398548    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:26.398712 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:26.398605    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:26.398712 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:26.398664    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:26.730000 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.729723    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7gmb"]
Jul 20 09:57:26.732598 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.732572    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qzr2"]
Jul 20 09:57:26.733152 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.733128    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5d4qt"]
Jul 20 09:57:26.770883 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.770846    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:26.879565 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:26.879536    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:27.183462 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:27.183436    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="3b4171ae314d6bac1f8b4c6c74be0417e455bfff2a0fccb0f7fd5149bee611c0" exitCode=0
Jul 20 09:57:27.183913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:27.183508    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:27.183913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:27.183537    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"3b4171ae314d6bac1f8b4c6c74be0417e455bfff2a0fccb0f7fd5149bee611c0"}
Jul 20 09:57:27.183913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:27.183598    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:27.183913 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:27.183714    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:27.184209 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:27.184185    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:27.184261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:27.184208    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:27.184409 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:27.184284    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:28.397871 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:28.397845    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:28.398188 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:28.397968    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:28.398188 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:28.398020    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:28.398188 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:28.398122    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:29.190268 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:29.190234    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="2cb3a6d9af187cf87b451321c95cbcceab4e4edd171ad42b50af5c91e20254eb" exitCode=0
Jul 20 09:57:29.190428 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:29.190284    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"2cb3a6d9af187cf87b451321c95cbcceab4e4edd171ad42b50af5c91e20254eb"}
Jul 20 09:57:29.397534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:29.397506    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:29.397681 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:29.397595    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:29.974690 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:29.974647    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.057337 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.057294    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.098049 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.098021    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:30.098198 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.098151    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:30.098249 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.098207    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret podName:f4142bda-92f5-4056-a55e-7d4bd77cd619 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:02.098188217 +0000 UTC m=+66.145612948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret") pod "global-pull-secret-syncer-q7gmb" (UID: "f4142bda-92f5-4056-a55e-7d4bd77cd619") : object "kube-system"/"original-pull-secret" not registered
Jul 20 09:57:30.158118 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.158051    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.199343 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.199316    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:30.199481 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.199374    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:30.199481 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199469    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:30.199565 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199481    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 09:57:30.199565 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199498    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 09:57:30.199565 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199510    2546 projected.go:196] Error preparing data for projected volume kube-api-access-vsqc9 for pod openshift-network-diagnostics/network-check-target-5d4qt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:30.199565 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199524    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:02.199510248 +0000 UTC m=+66.246934975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 09:57:30.199565 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.199557    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9 podName:c40010c2-e8eb-498e-94ae-e59f52a4fa6f nodeName:}" failed. No retries permitted until 2026-07-20 09:58:02.199541069 +0000 UTC m=+66.246965795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vsqc9" (UniqueName: "kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9") pod "network-check-target-5d4qt" (UID: "c40010c2-e8eb-498e-94ae-e59f52a4fa6f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 09:57:30.398227 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.398198    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:30.398408 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.398346    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5qzr2" podUID="44618c9c-a6c5-4e94-aedf-cadacba06315"
Jul 20 09:57:30.398408 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.398356    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:30.398491 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:30.398439    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5d4qt" podUID="c40010c2-e8eb-498e-94ae-e59f52a4fa6f"
Jul 20 09:57:30.486924 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.486891    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.564775 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.564729    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.757787 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.757707    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.870995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.870964    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:30.969451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:30.969411    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.069947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.069843    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.372972 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.372937    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.397353 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.397330    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:31.397495 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:31.397461    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q7gmb" podUID="f4142bda-92f5-4056-a55e-7d4bd77cd619"
Jul 20 09:57:31.477641 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.477600    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:31.804241 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.804157    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-155.ec2.internal" event="NodeReady"
Jul 20 09:57:31.804389 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.804313    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 09:57:31.913947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.913888    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94j7v"]
Jul 20 09:57:31.916626 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.916600    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:31.920815 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.920793    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 09:57:31.921060 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.921046    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txv2k\""
Jul 20 09:57:31.921404 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.921386    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 09:57:31.932492 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.932472    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xwfhn"]
Jul 20 09:57:31.934203 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.934184    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94j7v"]
Jul 20 09:57:31.934304 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.934280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:31.937529 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.937512    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 09:57:31.938161 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.938144    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 09:57:31.938260 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.938217    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 09:57:31.938559 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.938540    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bk72q\""
Jul 20 09:57:31.942465 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:31.942443    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwfhn"]
Jul 20 09:57:32.011390 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011363    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-cert\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.011390 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011391    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/2eb3e353-a052-4603-917b-754e8fef71b2-kube-api-access-mbv47\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.011595 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011408    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb3e353-a052-4603-917b-754e8fef71b2-tmp-dir\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.011595 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011427    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.011595 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011516    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsrj\" (UniqueName: \"kubernetes.io/projected/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-kube-api-access-zqsrj\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.011713 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.011595    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb3e353-a052-4603-917b-754e8fef71b2-config-volume\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112039 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112007    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-cert\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112047    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/2eb3e353-a052-4603-917b-754e8fef71b2-kube-api-access-mbv47\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112081    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb3e353-a052-4603-917b-754e8fef71b2-tmp-dir\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112161    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsrj\" (UniqueName: \"kubernetes.io/projected/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-kube-api-access-zqsrj\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:32.112195    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112233    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb3e353-a052-4603-917b-754e8fef71b2-config-volume\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:32.112285    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:32.612265343 +0000 UTC m=+36.659690071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:32.112451 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112403    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb3e353-a052-4603-917b-754e8fef71b2-tmp-dir\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.112997 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.112977    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2eb3e353-a052-4603-917b-754e8fef71b2-config-volume\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.116656 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.116533    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-cert\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.132929 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.132909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv47\" (UniqueName: \"kubernetes.io/projected/2eb3e353-a052-4603-917b-754e8fef71b2-kube-api-access-mbv47\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.133698 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.133678    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsrj\" (UniqueName: \"kubernetes.io/projected/47a080b5-3e17-4b8f-8f0c-b35f5655bde7-kube-api-access-zqsrj\") pod \"ingress-canary-xwfhn\" (UID: \"47a080b5-3e17-4b8f-8f0c-b35f5655bde7\") " pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.143278 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.143252    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.243704 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.243676    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xwfhn"
Jul 20 09:57:32.248551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.248501    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:32.398016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.397928    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:57:32.398016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.397974    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:57:32.401194 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.401165    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:57:32.402340 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.402306    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:57:32.402340 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.402319    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:57:32.402340 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.402333    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7v24\""
Jul 20 09:57:32.402583 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.402319    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f4r9z\""
Jul 20 09:57:32.614718 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:32.614677    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:32.614906 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:32.614840    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:32.614967 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:32.614926    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:33.614902634 +0000 UTC m=+37.662327357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:33.258023 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:33.257994    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xwfhn"]
Jul 20 09:57:33.262180 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:57:33.262155    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a080b5_3e17_4b8f_8f0c_b35f5655bde7.slice/crio-25d756505239b9c9319e8170c386d9d6475ce9b8a984ac9d733c342c30ca89a3 WatchSource:0}: Error finding container 25d756505239b9c9319e8170c386d9d6475ce9b8a984ac9d733c342c30ca89a3: Status 404 returned error can't find the container with id 25d756505239b9c9319e8170c386d9d6475ce9b8a984ac9d733c342c30ca89a3
Jul 20 09:57:33.397974 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:33.397909    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:57:33.401033 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:33.401015    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:57:33.620455 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:33.620426    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:33.620604 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:33.620574    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:33.620650 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:33.620631    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:35.620617431 +0000 UTC m=+39.668042153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:34.203858 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:34.203819    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwfhn" event={"ID":"47a080b5-3e17-4b8f-8f0c-b35f5655bde7","Type":"ContainerStarted","Data":"25d756505239b9c9319e8170c386d9d6475ce9b8a984ac9d733c342c30ca89a3"}
Jul 20 09:57:34.207083 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:34.207038    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="fdeb6980104b338282a0aed8ade60ab3ee04bb5d4d3997b1afb18c464a9b4177" exitCode=0
Jul 20 09:57:34.207219 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:34.207107    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"fdeb6980104b338282a0aed8ade60ab3ee04bb5d4d3997b1afb18c464a9b4177"}
Jul 20 09:57:34.351827 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:34.351796    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:35.213838 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:35.213801    2546 generic.go:356] "Generic (PLEG): container finished" podID="2578aede-60eb-4f71-94ed-06e90d6cf997" containerID="d8f4c56493e2af81c0467ea86982902a3252fa709ac9016298393dbd90c68ec2" exitCode=0
Jul 20 09:57:35.214027 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:35.213868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerDied","Data":"d8f4c56493e2af81c0467ea86982902a3252fa709ac9016298393dbd90c68ec2"}
Jul 20 09:57:35.634547 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:35.634497    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:35.635040 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:35.634667    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:35.635040 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:35.634761    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:39.634740058 +0000 UTC m=+43.682164794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:36.219721 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.219683    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xwfhn" event={"ID":"47a080b5-3e17-4b8f-8f0c-b35f5655bde7","Type":"ContainerStarted","Data":"c23167a280274de52553f95fe58a969b9022d94e13f41d9bd7afd9c464079cdc"}
Jul 20 09:57:36.222911 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.222884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" event={"ID":"2578aede-60eb-4f71-94ed-06e90d6cf997","Type":"ContainerStarted","Data":"ed1e4d0220c96583ef2822cf972f15b2f584b371bc03552221130a4b1e2d44d1"}
Jul 20 09:57:36.240691 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.240654    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xwfhn" podStartSLOduration=3.054120689 podStartE2EDuration="5.240640674s" totalImagesPullingTime="2.186519985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.240293155 +0000 UTC m=+40.287717900" watchObservedRunningTime="2026-07-20 09:57:36.240640674 +0000 UTC m=+40.288065397"
Jul 20 09:57:36.266037 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.265989    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ctm7j" podStartSLOduration=9.408700212 podStartE2EDuration="40.265970503s" totalImagesPullingTime="30.857270291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:57:36.264147713 +0000 UTC m=+40.311572458" watchObservedRunningTime="2026-07-20 09:57:36.265970503 +0000 UTC m=+40.313395252"
Jul 20 09:57:36.400890 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.400855    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:36.504360 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.504328    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:36.935057 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:36.935030    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:37.033511 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:37.033484    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:39.661987 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:39.661947    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:39.662389 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:39.662120    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:39.662389 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:39.662184    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:57:47.662167412 +0000 UTC m=+51.709592135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:43.350384 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:43.350344    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:44.719283 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:44.719248    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.153303 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:46.153266    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.222122 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:46.222092    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.451822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:46.451740    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:46.524087 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:46.524035    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:47.716716 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:47.716671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:57:47.717172 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:47.716816    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:57:47.717247 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:57:47.717182    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:03.716862016 +0000 UTC m=+67.764286739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:57:52.757872 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:52.757840    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:52.825677 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:52.825645    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:58.207339 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:58.207311    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cqlg8"
Jul 20 09:57:58.367416 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:58.367381    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:57:58.467198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:57:58.467112    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:02.115543 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.115506    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:58:02.118970 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.118952    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 09:58:02.129747 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.129715    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4142bda-92f5-4056-a55e-7d4bd77cd619-original-pull-secret\") pod \"global-pull-secret-syncer-q7gmb\" (UID: \"f4142bda-92f5-4056-a55e-7d4bd77cd619\") " pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:58:02.206395 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.206372    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q7gmb"
Jul 20 09:58:02.216569 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.216542    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:58:02.216668 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.216592    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:58:02.219659 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.219640    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 09:58:02.219757 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.219703    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 09:58:02.227534 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:02.227513    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 09:58:02.227630 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:02.227570    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs podName:44618c9c-a6c5-4e94-aedf-cadacba06315 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:06.227554003 +0000 UTC m=+130.274978725 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs") pod "network-metrics-daemon-5qzr2" (UID: "44618c9c-a6c5-4e94-aedf-cadacba06315") : secret "metrics-daemon-secret" not found
Jul 20 09:58:02.230202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.230185    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 09:58:02.241364 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.241333    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqc9\" (UniqueName: \"kubernetes.io/projected/c40010c2-e8eb-498e-94ae-e59f52a4fa6f-kube-api-access-vsqc9\") pod \"network-check-target-5d4qt\" (UID: \"c40010c2-e8eb-498e-94ae-e59f52a4fa6f\") " pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:58:02.328409 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.328372    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q7gmb"]
Jul 20 09:58:02.420935 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.420879    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f4r9z\""
Jul 20 09:58:02.426637 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.426622    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:58:02.546465 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:02.546441    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5d4qt"]
Jul 20 09:58:02.550368 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:02.549145    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40010c2_e8eb_498e_94ae_e59f52a4fa6f.slice/crio-69e337b8ed0b1ec118325db22210d63cf335fbec951d0cd77bb44db034654d9d WatchSource:0}: Error finding container 69e337b8ed0b1ec118325db22210d63cf335fbec951d0cd77bb44db034654d9d: Status 404 returned error can't find the container with id 69e337b8ed0b1ec118325db22210d63cf335fbec951d0cd77bb44db034654d9d
Jul 20 09:58:03.285145 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:03.285054    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5d4qt" event={"ID":"c40010c2-e8eb-498e-94ae-e59f52a4fa6f","Type":"ContainerStarted","Data":"69e337b8ed0b1ec118325db22210d63cf335fbec951d0cd77bb44db034654d9d"}
Jul 20 09:58:03.286354 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:03.286298    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q7gmb" event={"ID":"f4142bda-92f5-4056-a55e-7d4bd77cd619","Type":"ContainerStarted","Data":"83052cd5e4dd1b2699437d17458a9cf0a3f728e23e3ed5300026b9e6884b7a02"}
Jul 20 09:58:03.726889 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:03.726848    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:58:03.727062 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:03.726997    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:58:03.727062 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:03.727061    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:35.727040745 +0000 UTC m=+99.774465488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:07.299195 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:07.299159    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5d4qt" event={"ID":"c40010c2-e8eb-498e-94ae-e59f52a4fa6f","Type":"ContainerStarted","Data":"dfc4768daabbcbd6a0bcb5aa6e9dd983a3b05a52adee79cb08eeb83ffa9e3aa9"}
Jul 20 09:58:07.299662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:07.299300    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:58:07.300682 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:07.300644    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q7gmb" event={"ID":"f4142bda-92f5-4056-a55e-7d4bd77cd619","Type":"ContainerStarted","Data":"5ea5f1d14c62718b5a1b677a8d4faced76cd7b95e8e4026898154ec3a203c859"}
Jul 20 09:58:07.320411 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:07.318785    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5d4qt" podStartSLOduration=66.790592664 podStartE2EDuration="1m11.318763471s" totalImagesPullingTime="4.528170807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:07.31754963 +0000 UTC m=+71.364974376" watchObservedRunningTime="2026-07-20 09:58:07.318763471 +0000 UTC m=+71.366188217"
Jul 20 09:58:07.333632 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:07.333591    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q7gmb" podStartSLOduration=66.582407146 podStartE2EDuration="1m11.333578642s" totalImagesPullingTime="4.751171496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:07.332719334 +0000 UTC m=+71.380144082" watchObservedRunningTime="2026-07-20 09:58:07.333578642 +0000 UTC m=+71.381003423"
Jul 20 09:58:08.091562 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:08.091525    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:08.167428 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:08.167396    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:23.108250 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.108129    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"]
Jul 20 09:58:23.110775 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.110755    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.116246 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.116219    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 20 09:58:23.116246 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.116229    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xsf2v\""
Jul 20 09:58:23.116410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.116300    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 09:58:23.116410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.116325    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 20 09:58:23.116410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.116317    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 09:58:23.122358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.122339    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"]
Jul 20 09:58:23.151925 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.151893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.152016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.151925    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmmbv\" (UniqueName: \"kubernetes.io/projected/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-kube-api-access-rmmbv\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.152016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.151954    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.252375 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.252352    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.252478 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.252381    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmmbv\" (UniqueName: \"kubernetes.io/projected/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-kube-api-access-rmmbv\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.252478 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.252399    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.252586 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:23.252487    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:23.252586 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:23.252542    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:23.752528091 +0000 UTC m=+87.799952815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:23.252973 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.252954    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.267403 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.267387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmmbv\" (UniqueName: \"kubernetes.io/projected/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-kube-api-access-rmmbv\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.312707 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.312685    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-c54fp"]
Jul 20 09:58:23.314687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.314675    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.317771 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.317752    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 20 09:58:23.317880 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.317858    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 20 09:58:23.317947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.317888    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6nsx2\""
Jul 20 09:58:23.318000 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.317945    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 09:58:23.318295 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.318275    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 09:58:23.323323 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.323305    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 20 09:58:23.328121 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.328103    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-c54fp"]
Jul 20 09:58:23.385511 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.385455    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:23.454056 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454033    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-service-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.454159 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nc24\" (UniqueName: \"kubernetes.io/projected/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-kube-api-access-6nc24\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.454198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454157    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-snapshots\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.454198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454188    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-tmp\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.454261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454203    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.454261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.454235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-serving-cert\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.486294 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.486274    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:23.554617 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554582    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-service-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.554706 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554631    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nc24\" (UniqueName: \"kubernetes.io/projected/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-kube-api-access-6nc24\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.554818 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554788    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-snapshots\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.554858 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554846    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-tmp\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.554973 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.555032 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.554999    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-serving-cert\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.555299 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.555274    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-service-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.555377 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.555325    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-tmp\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.555419 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.555375    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-snapshots\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.555661 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.555644    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.557280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.557256    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-serving-cert\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.569709 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.569687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nc24\" (UniqueName: \"kubernetes.io/projected/2b0979e2-1e37-4d9a-a017-3771bf4a5d27-kube-api-access-6nc24\") pod \"insights-operator-555d9d7885-c54fp\" (UID: \"2b0979e2-1e37-4d9a-a017-3771bf4a5d27\") " pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.623677 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.623663    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-c54fp"
Jul 20 09:58:23.741159 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.741122    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-c54fp"]
Jul 20 09:58:23.744385 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:23.744357    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0979e2_1e37_4d9a_a017_3771bf4a5d27.slice/crio-7201a126157911e1bc878267d4f2c4b77199d63cb36c774ae61c42462d77d250 WatchSource:0}: Error finding container 7201a126157911e1bc878267d4f2c4b77199d63cb36c774ae61c42462d77d250: Status 404 returned error can't find the container with id 7201a126157911e1bc878267d4f2c4b77199d63cb36c774ae61c42462d77d250
Jul 20 09:58:23.755993 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:23.755947    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:23.756134 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:23.756117    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:23.756194 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:23.756184    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:24.756165543 +0000 UTC m=+88.803590282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:24.342184 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:24.342144    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-c54fp" event={"ID":"2b0979e2-1e37-4d9a-a017-3771bf4a5d27","Type":"ContainerStarted","Data":"7201a126157911e1bc878267d4f2c4b77199d63cb36c774ae61c42462d77d250"}
Jul 20 09:58:24.764046 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:24.763968    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:24.764217 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:24.764092    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:24.764217 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:24.764146    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:26.764133055 +0000 UTC m=+90.811557779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:26.350755 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.350721    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-c54fp" event={"ID":"2b0979e2-1e37-4d9a-a017-3771bf4a5d27","Type":"ContainerStarted","Data":"7cd5e0d743022e76da2788f778df1913cfd554b2e16604e33d7e262251e6773d"}
Jul 20 09:58:26.371205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.371159    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-c54fp" podStartSLOduration=1.6991992489999999 podStartE2EDuration="3.371145507s" totalImagesPullingTime="1.671946258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:26.369688988 +0000 UTC m=+90.417113747" watchObservedRunningTime="2026-07-20 09:58:26.371145507 +0000 UTC m=+90.418570252"
Jul 20 09:58:26.585914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.585881    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-npw5t"]
Jul 20 09:58:26.588471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.588445    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.591284 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.591266    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 09:58:26.591832 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.591798    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wds68\""
Jul 20 09:58:26.591832 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.591813    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 09:58:26.607332 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.607281    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-npw5t"]
Jul 20 09:58:26.618210 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.618181    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:26.678002 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.677972    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a1b7fa-4991-419f-9bdd-6f474c315059-data-volume\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.678153 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.678013    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.678153 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.678041    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a1b7fa-4991-419f-9bdd-6f474c315059-crio-socket\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.678353 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.678174    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.678353 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.678199    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqth\" (UniqueName: \"kubernetes.io/projected/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-api-access-pxqth\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.709020 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.708992    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:26.779470 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779447    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a1b7fa-4991-419f-9bdd-6f474c315059-crio-socket\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779580 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:26.779580 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779559    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94a1b7fa-4991-419f-9bdd-6f474c315059-crio-socket\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779615    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqth\" (UniqueName: \"kubernetes.io/projected/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-api-access-pxqth\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779621    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779642    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a1b7fa-4991-419f-9bdd-6f474c315059-data-volume\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779657    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779663    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.779687 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779684    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:27.279665522 +0000 UTC m=+91.327090262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:26.779991 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779727    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:30.779715837 +0000 UTC m=+94.827140562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:26.779991 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779733    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:26.779991 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:26.779824    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:27.279807181 +0000 UTC m=+91.327231909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:26.779991 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.779941    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a1b7fa-4991-419f-9bdd-6f474c315059-data-volume\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:26.790336 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:26.790313    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqth\" (UniqueName: \"kubernetes.io/projected/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-api-access-pxqth\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:27.283551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:27.283515    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:27.283753 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:27.283559    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:27.283753 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:27.283648    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:27.283753 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:27.283663    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:27.283753 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:27.283717    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.283699668 +0000 UTC m=+92.331124392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:27.283753 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:27.283739    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:28.283724642 +0000 UTC m=+92.331149383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:28.291554 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:28.291518    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:28.291929 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:28.291563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:28.291929 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:28.291638    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:28.291929 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:28.291684    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:28.291929 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:28.291708    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:30.291690365 +0000 UTC m=+94.339115089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:28.291929 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:28.291724    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:30.291717771 +0000 UTC m=+94.339142493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:28.362552 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:28.362531    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4l429_df265ea5-1894-4478-8759-17110f267128/dns-node-resolver/0.log"
Jul 20 09:58:29.160245 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:29.160220    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22smx_742d5ccd-da6c-4f25-b376-9d9b85e9cee1/node-ca/0.log"
Jul 20 09:58:30.305605 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:30.305564    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:30.305605 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:30.305613    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:30.306037 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.305683    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:30.306037 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.305702    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:30.306037 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.305746    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:34.305730779 +0000 UTC m=+98.353155505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:30.306037 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.305760    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:34.305754042 +0000 UTC m=+98.353178765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:30.360610 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:30.360586    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwfhn_47a080b5-3e17-4b8f-8f0c-b35f5655bde7/serve-healthcheck-canary/0.log"
Jul 20 09:58:30.808905 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:30.808872    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:30.809099 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.809017    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:30.809149 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:30.809101    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:38.809080725 +0000 UTC m=+102.856505463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:32.812986 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.812950    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"]
Jul 20 09:58:32.817479 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.817452    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:32.821100 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.821060    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 20 09:58:32.821221 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.821191    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 20 09:58:32.821478 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.821458    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:58:32.822380 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.822355    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 20 09:58:32.822477 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.822446    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qsrr2\""
Jul 20 09:58:32.832614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.832593    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"]
Jul 20 09:58:32.923266 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.923242    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:32.923358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.923283    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwbmc\" (UniqueName: \"kubernetes.io/projected/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-kube-api-access-wwbmc\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:32.923358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.923304    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-config\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:32.940442 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.940415    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"]
Jul 20 09:58:32.942240 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.942227    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"
Jul 20 09:58:32.946261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.946240    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 20 09:58:32.946725 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.946703    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:58:32.946877 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.946770    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l6hlb\""
Jul 20 09:58:32.948534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.948511    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"]
Jul 20 09:58:32.950713 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.950698    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:32.954109 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.954093    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 20 09:58:32.954369 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.954351    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 20 09:58:32.954494 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.954480    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:58:32.954617 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.954603    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mtlwd\""
Jul 20 09:58:32.959025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.959005    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-pcqb5"]
Jul 20 09:58:32.960696 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.960684    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:32.961965 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.961947    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"]
Jul 20 09:58:32.964809 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.964793    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 20 09:58:32.965259 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.965233    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 20 09:58:32.965350 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.965277    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 20 09:58:32.965350 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.965239    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 20 09:58:32.965608 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.965588    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gcb5l\""
Jul 20 09:58:32.970217 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.970188    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"]
Jul 20 09:58:32.976118 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:32.976101    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 20 09:58:33.000189 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.000171    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-pcqb5"]
Jul 20 09:58:33.023853 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.023823    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.023947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.023856    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mr9\" (UniqueName: \"kubernetes.io/projected/294fe74b-4e25-4eff-bd9b-c0bff79829a9-kube-api-access-96mr9\") pod \"volume-data-source-validator-78cf4c6f6-phmdw\" (UID: \"294fe74b-4e25-4eff-bd9b-c0bff79829a9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"
Jul 20 09:58:33.023947 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.023883    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-trusted-ca\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.024036 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.023955    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.024036 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.023984    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-config\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.024036 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024017    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-config\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.024201 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024118    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwbmc\" (UniqueName: \"kubernetes.io/projected/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-kube-api-access-wwbmc\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.024255 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfp6c\" (UniqueName: \"kubernetes.io/projected/138c267a-c727-41ce-9322-1b88cd7153b6-kube-api-access-pfp6c\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.024315 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024286    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19a58fee-11c7-436b-8da3-648634b1f44c-serving-cert\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.024383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024363    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwthq\" (UniqueName: \"kubernetes.io/projected/19a58fee-11c7-436b-8da3-648634b1f44c-kube-api-access-hwthq\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.024551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.024535    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-config\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.026026 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.026008    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-serving-cert\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.042137 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.042115    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwbmc\" (UniqueName: \"kubernetes.io/projected/a02fa1cc-3cf8-46d8-bf8d-351f97251ded-kube-api-access-wwbmc\") pod \"service-ca-operator-5bcbc94c78-tzcfj\" (UID: \"a02fa1cc-3cf8-46d8-bf8d-351f97251ded\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.124662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfp6c\" (UniqueName: \"kubernetes.io/projected/138c267a-c727-41ce-9322-1b88cd7153b6-kube-api-access-pfp6c\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.124751 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124676    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19a58fee-11c7-436b-8da3-648634b1f44c-serving-cert\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.124751 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124708    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwthq\" (UniqueName: \"kubernetes.io/projected/19a58fee-11c7-436b-8da3-648634b1f44c-kube-api-access-hwthq\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.124751 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124738    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.124863 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124763    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mr9\" (UniqueName: \"kubernetes.io/projected/294fe74b-4e25-4eff-bd9b-c0bff79829a9-kube-api-access-96mr9\") pod \"volume-data-source-validator-78cf4c6f6-phmdw\" (UID: \"294fe74b-4e25-4eff-bd9b-c0bff79829a9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"
Jul 20 09:58:33.124863 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-trusted-ca\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.124948 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.124864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-config\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.124948 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:33.124884    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:58:33.125043 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:33.124957    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls podName:138c267a-c727-41ce-9322-1b88cd7153b6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:33.624938394 +0000 UTC m=+97.672363120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7dw94" (UID: "138c267a-c727-41ce-9322-1b88cd7153b6") : secret "samples-operator-tls" not found
Jul 20 09:58:33.125566 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.125548    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-config\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.125661 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.125644    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19a58fee-11c7-436b-8da3-648634b1f44c-trusted-ca\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.126162 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.126151    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"
Jul 20 09:58:33.126748 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.126729    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19a58fee-11c7-436b-8da3-648634b1f44c-serving-cert\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.135202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.135181    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwthq\" (UniqueName: \"kubernetes.io/projected/19a58fee-11c7-436b-8da3-648634b1f44c-kube-api-access-hwthq\") pod \"console-operator-565b7b569d-pcqb5\" (UID: \"19a58fee-11c7-436b-8da3-648634b1f44c\") " pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.137943 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.137918    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfp6c\" (UniqueName: \"kubernetes.io/projected/138c267a-c727-41ce-9322-1b88cd7153b6-kube-api-access-pfp6c\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.138324 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.138305    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mr9\" (UniqueName: \"kubernetes.io/projected/294fe74b-4e25-4eff-bd9b-c0bff79829a9-kube-api-access-96mr9\") pod \"volume-data-source-validator-78cf4c6f6-phmdw\" (UID: \"294fe74b-4e25-4eff-bd9b-c0bff79829a9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"
Jul 20 09:58:33.251192 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.251164    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"
Jul 20 09:58:33.255472 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.255447    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj"]
Jul 20 09:58:33.258548 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:33.258524    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02fa1cc_3cf8_46d8_bf8d_351f97251ded.slice/crio-7c066a527f4c79c08653982db60f902dcfb8dcb95c359026d1aa8e194b6c2526 WatchSource:0}: Error finding container 7c066a527f4c79c08653982db60f902dcfb8dcb95c359026d1aa8e194b6c2526: Status 404 returned error can't find the container with id 7c066a527f4c79c08653982db60f902dcfb8dcb95c359026d1aa8e194b6c2526
Jul 20 09:58:33.269737 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.269716    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:33.367211 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.367184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj" event={"ID":"a02fa1cc-3cf8-46d8-bf8d-351f97251ded","Type":"ContainerStarted","Data":"7c066a527f4c79c08653982db60f902dcfb8dcb95c359026d1aa8e194b6c2526"}
Jul 20 09:58:33.376687 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.376633    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw"]
Jul 20 09:58:33.379218 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:33.379190    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294fe74b_4e25_4eff_bd9b_c0bff79829a9.slice/crio-a192f1b994a79d6b2f24f0345cf7d6bf81dc89cc15563627f1f241c6e650e2d7 WatchSource:0}: Error finding container a192f1b994a79d6b2f24f0345cf7d6bf81dc89cc15563627f1f241c6e650e2d7: Status 404 returned error can't find the container with id a192f1b994a79d6b2f24f0345cf7d6bf81dc89cc15563627f1f241c6e650e2d7
Jul 20 09:58:33.408326 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.408304    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-pcqb5"]
Jul 20 09:58:33.412039 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:33.412012    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a58fee_11c7_436b_8da3_648634b1f44c.slice/crio-64cf805598421cf79e77c01fc7951e02677d616fe434836cff019130a8b07ea0 WatchSource:0}: Error finding container 64cf805598421cf79e77c01fc7951e02677d616fe434836cff019130a8b07ea0: Status 404 returned error can't find the container with id 64cf805598421cf79e77c01fc7951e02677d616fe434836cff019130a8b07ea0
Jul 20 09:58:33.628791 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:33.628716    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:33.628931 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:33.628876    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:58:33.628984 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:33.628947    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls podName:138c267a-c727-41ce-9322-1b88cd7153b6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:34.628923243 +0000 UTC m=+98.676347966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7dw94" (UID: "138c267a-c727-41ce-9322-1b88cd7153b6") : secret "samples-operator-tls" not found
Jul 20 09:58:34.335282 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:34.335187    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:34.335282 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:34.335285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:34.335779 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.335355    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:34.335779 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.335355    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 09:58:34.335779 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.335437    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:42.335416562 +0000 UTC m=+106.382841298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:34.335779 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.335488    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:42.335472713 +0000 UTC m=+106.382897443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : configmap "kube-rbac-proxy" not found
Jul 20 09:58:34.371999 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:34.371897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" event={"ID":"19a58fee-11c7-436b-8da3-648634b1f44c","Type":"ContainerStarted","Data":"64cf805598421cf79e77c01fc7951e02677d616fe434836cff019130a8b07ea0"}
Jul 20 09:58:34.373338 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:34.373305    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw" event={"ID":"294fe74b-4e25-4eff-bd9b-c0bff79829a9","Type":"ContainerStarted","Data":"a192f1b994a79d6b2f24f0345cf7d6bf81dc89cc15563627f1f241c6e650e2d7"}
Jul 20 09:58:34.638570 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:34.638485    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:34.638733 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.638647    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:58:34.638733 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:34.638723    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls podName:138c267a-c727-41ce-9322-1b88cd7153b6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:36.638703256 +0000 UTC m=+100.686127978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7dw94" (UID: "138c267a-c727-41ce-9322-1b88cd7153b6") : secret "samples-operator-tls" not found
Jul 20 09:58:35.378109 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:35.378061    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw" event={"ID":"294fe74b-4e25-4eff-bd9b-c0bff79829a9","Type":"ContainerStarted","Data":"bef852909af7c980e42a4e8b652ebb4099577ff5e24439652899f748da8d9769"}
Jul 20 09:58:35.401627 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:35.401575    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-phmdw" podStartSLOduration=2.337102996 podStartE2EDuration="3.401560649s" totalImagesPullingTime="1.064457653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:35.400871365 +0000 UTC m=+99.448296111" watchObservedRunningTime="2026-07-20 09:58:35.401560649 +0000 UTC m=+99.448985393"
Jul 20 09:58:35.594049 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:35.593598    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:35.690460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:35.690401    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:35.749633 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:35.749607    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:58:35.749782 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:35.749765    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 09:58:35.749864 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:35.749851    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls podName:2eb3e353-a052-4603-917b-754e8fef71b2 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:39.749828348 +0000 UTC m=+163.797253075 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls") pod "dns-default-94j7v" (UID: "2eb3e353-a052-4603-917b-754e8fef71b2") : secret "dns-default-metrics-tls" not found
Jul 20 09:58:36.093059 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.093021    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:36.292305 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.292269    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:36.382955 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.382888    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/0.log"
Jul 20 09:58:36.382955 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.382924    2546 generic.go:356] "Generic (PLEG): container finished" podID="19a58fee-11c7-436b-8da3-648634b1f44c" containerID="73121bdb9a89a5668b1d370683657a42b4f266bbf1cb226bca6d1f0829f8e444" exitCode=255
Jul 20 09:58:36.383424 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.383023    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" event={"ID":"19a58fee-11c7-436b-8da3-648634b1f44c","Type":"ContainerDied","Data":"73121bdb9a89a5668b1d370683657a42b4f266bbf1cb226bca6d1f0829f8e444"}
Jul 20 09:58:36.383424 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.383223    2546 scope.go:117] "RemoveContainer" containerID="73121bdb9a89a5668b1d370683657a42b4f266bbf1cb226bca6d1f0829f8e444"
Jul 20 09:58:36.384508 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.384486    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj" event={"ID":"a02fa1cc-3cf8-46d8-bf8d-351f97251ded","Type":"ContainerStarted","Data":"90b3365aec2c46f8cad56268ae3686771bdafdeddaada4998d9f5532327fdf0e"}
Jul 20 09:58:36.397220 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.397194    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:36.427300 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.427254    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj" podStartSLOduration=2.182076432 podStartE2EDuration="4.427235606s" totalImagesPullingTime="2.245159174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:36.426201217 +0000 UTC m=+100.473625975" watchObservedRunningTime="2026-07-20 09:58:36.427235606 +0000 UTC m=+100.474660353"
Jul 20 09:58:36.656441 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.656357    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:36.656575 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:36.656479    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:58:36.656575 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:36.656562    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls podName:138c267a-c727-41ce-9322-1b88cd7153b6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:40.656520315 +0000 UTC m=+104.703945055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7dw94" (UID: "138c267a-c727-41ce-9322-1b88cd7153b6") : secret "samples-operator-tls" not found
Jul 20 09:58:36.832045 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.832011    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:36.932016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:36.931946    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:37.251473 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.251383    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:37.389145 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389117    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 09:58:37.389530 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389432    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/0.log"
Jul 20 09:58:37.389530 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389459    2546 generic.go:356] "Generic (PLEG): container finished" podID="19a58fee-11c7-436b-8da3-648634b1f44c" containerID="7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d" exitCode=255
Jul 20 09:58:37.389596 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" event={"ID":"19a58fee-11c7-436b-8da3-648634b1f44c","Type":"ContainerDied","Data":"7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d"}
Jul 20 09:58:37.389631 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389608    2546 scope.go:117] "RemoveContainer" containerID="73121bdb9a89a5668b1d370683657a42b4f266bbf1cb226bca6d1f0829f8e444"
Jul 20 09:58:37.389824 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.389801    2546 scope.go:117] "RemoveContainer" containerID="7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d"
Jul 20 09:58:37.389994 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:37.389972    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-pcqb5_openshift-console-operator(19a58fee-11c7-436b-8da3-648634b1f44c)\"" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" podUID="19a58fee-11c7-436b-8da3-648634b1f44c"
Jul 20 09:58:37.565011 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.564933    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:37.746666 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:37.746633    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:38.305505 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:38.305477    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5d4qt"
Jul 20 09:58:38.393955 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:38.393929    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 09:58:38.394344 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:38.394242    2546 scope.go:117] "RemoveContainer" containerID="7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d"
Jul 20 09:58:38.394416 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:38.394395    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-pcqb5_openshift-console-operator(19a58fee-11c7-436b-8da3-648634b1f44c)\"" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" podUID="19a58fee-11c7-436b-8da3-648634b1f44c"
Jul 20 09:58:38.873388 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:38.873355    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:38.873533 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:38.873495    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:38.873575 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:38.873564    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls podName:7be60d37-9c23-4eb3-bfcc-c5688f5ee538 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:54.873548295 +0000 UTC m=+118.920973022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-m5ggr" (UID: "7be60d37-9c23-4eb3-bfcc-c5688f5ee538") : secret "cluster-monitoring-operator-tls" not found
Jul 20 09:58:38.919911 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:38.919874    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.017682 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:39.017656    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.265300 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:39.265226    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.394522 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:39.394490    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:39.853937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:39.853905    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:40.686767 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:40.686728    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:40.687263 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:40.686860    2546 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 20 09:58:40.687263 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:40.686918    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls podName:138c267a-c727-41ce-9322-1b88cd7153b6 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:48.686903686 +0000 UTC m=+112.734328408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-7dw94" (UID: "138c267a-c727-41ce-9322-1b88cd7153b6") : secret "samples-operator-tls" not found
Jul 20 09:58:40.855878 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:40.855846    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:42.399311 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:42.399273    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:42.399715 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:42.399373    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:42.399715 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:42.399490    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:42.399715 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:42.399557    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls podName:94a1b7fa-4991-419f-9bdd-6f474c315059 nodeName:}" failed. No retries permitted until 2026-07-20 09:58:58.399539284 +0000 UTC m=+122.446964011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls") pod "insights-runtime-extractor-npw5t" (UID: "94a1b7fa-4991-419f-9bdd-6f474c315059") : secret "insights-runtime-extractor-tls" not found
Jul 20 09:58:42.399905 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:42.399881    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94a1b7fa-4991-419f-9bdd-6f474c315059-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:43.013002 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.012965    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.195259 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.195217    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.270281 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.270223    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:43.270281 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.270268    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:43.270602 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.270591    2546 scope.go:117] "RemoveContainer" containerID="7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d"
Jul 20 09:58:43.270751 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:58:43.270734    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-pcqb5_openshift-console-operator(19a58fee-11c7-436b-8da3-648634b1f44c)\"" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" podUID="19a58fee-11c7-436b-8da3-648634b1f44c"
Jul 20 09:58:43.296212 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.296190    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.397499 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.397477    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.495371 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.495351    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:43.787903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:43.787874    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:44.952437 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:44.952397    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.051439 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.051413    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.135479 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.135444    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.318106 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.318006    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.521965 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.521931    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.666374 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.666344    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:45.919171 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:45.919097    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.735922 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:46.735892    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.900836 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:46.900804    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:46.985428 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:46.985400    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.152417 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:47.152388    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.319381 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:47.319344    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.393693 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:47.393660    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.517800 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:47.517714    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:47.739881 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:47.739848    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.748177 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:48.748142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:48.750371 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:48.750348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/138c267a-c727-41ce-9322-1b88cd7153b6-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-7dw94\" (UID: \"138c267a-c727-41ce-9322-1b88cd7153b6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:48.859318 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:48.859289    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"
Jul 20 09:58:48.945558 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:48.945523    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:48.981530 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:48.981495    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94"]
Jul 20 09:58:49.048881 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.048851    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.146989 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.146960    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.311015 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.310935    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.426584 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.426556    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94" event={"ID":"138c267a-c727-41ce-9322-1b88cd7153b6","Type":"ContainerStarted","Data":"3e1354e49e71c2d7cc22c21f30a1a2acd0fe850f11a6778d317c41ea348bc775"}
Jul 20 09:58:49.517704 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.517675    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:49.719320 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:49.719292    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:50.508938 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:50.508883    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.110360 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.110327    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.320086 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.319971    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.433991 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.433903    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94" event={"ID":"138c267a-c727-41ce-9322-1b88cd7153b6","Type":"ContainerStarted","Data":"653c495c30d9f8c186bdad3f5a00e6cdf1bff0e84ccf073760ee5275b4052722"}
Jul 20 09:58:51.433991 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.433954    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94" event={"ID":"138c267a-c727-41ce-9322-1b88cd7153b6","Type":"ContainerStarted","Data":"d04592391f3527992607e77fab7212881df6baf1d3719ed21d88451b973e8637"}
Jul 20 09:58:51.455015 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.454971    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-7dw94" podStartSLOduration=18.009384402 podStartE2EDuration="19.45495938s" totalImagesPullingTime="1.445574978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:51.45445412 +0000 UTC m=+115.501879060" watchObservedRunningTime="2026-07-20 09:58:51.45495938 +0000 UTC m=+115.502384124"
Jul 20 09:58:51.512010 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.511982    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.899289 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.899260    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:51.968038 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:51.968015    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:52.357507 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:52.357471    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:52.906374 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:52.906340    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.306869 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:53.306779    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.516378 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:53.516322    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:53.716560 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:53.716530    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:54.398478 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:54.398455    2546 scope.go:117] "RemoveContainer" containerID="7cf40972738fe8f5a141a593162cf58d1a9a81864ebdd7f760a9b069ed2e9a1d"
Jul 20 09:58:54.893304 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:54.893262    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:54.895957 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:54.895927    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7be60d37-9c23-4eb3-bfcc-c5688f5ee538-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-m5ggr\" (UID: \"7be60d37-9c23-4eb3-bfcc-c5688f5ee538\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:54.919911 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:54.919888    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"
Jul 20 09:58:54.938515 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:54.938488    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.038443 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.038408    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.042645 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.042623    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr"]
Jul 20 09:58:55.045474 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:55.045436    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be60d37_9c23_4eb3_bfcc_c5688f5ee538.slice/crio-4e759e5cf9f1f58fbd55406a8d182765e479da8684815f33d360bee27c41d4ca WatchSource:0}: Error finding container 4e759e5cf9f1f58fbd55406a8d182765e479da8684815f33d360bee27c41d4ca: Status 404 returned error can't find the container with id 4e759e5cf9f1f58fbd55406a8d182765e479da8684815f33d360bee27c41d4ca
Jul 20 09:58:55.138522 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.138497    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.266249 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.266173    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.448407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.448380    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 09:58:55.448997 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.448492    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" event={"ID":"19a58fee-11c7-436b-8da3-648634b1f44c","Type":"ContainerStarted","Data":"e3b4ba6cafb01a0863f4db2d153284b06fb4e49014b1027e0cf3c796db8c837a"}
Jul 20 09:58:55.448997 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.448786    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:55.449650 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.449625    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr" event={"ID":"7be60d37-9c23-4eb3-bfcc-c5688f5ee538","Type":"ContainerStarted","Data":"4e759e5cf9f1f58fbd55406a8d182765e479da8684815f33d360bee27c41d4ca"}
Jul 20 09:58:55.453420 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.453400    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5"
Jul 20 09:58:55.484913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.484890    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.518996 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.518923    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-pcqb5" podStartSLOduration=21.426440456 podStartE2EDuration="23.518911035s" totalImagesPullingTime="2.092470579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:55.483907316 +0000 UTC m=+119.531332061" watchObservedRunningTime="2026-07-20 09:58:55.518911035 +0000 UTC m=+119.566335779"
Jul 20 09:58:55.606871 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.606836    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:55.711116 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:55.711054    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:56.826865 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:56.826825    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:56.927252 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:56.927179    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:57.027028 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.026975    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:57.128871 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.128833    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:57.226801 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.226721    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:57.298761 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.298729    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:57.457393 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.457362    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr" event={"ID":"7be60d37-9c23-4eb3-bfcc-c5688f5ee538","Type":"ContainerStarted","Data":"d9e241c9ef59360301c8dc10a1a5be0473b04371586e1389a013fae26f2cbb43"}
Jul 20 09:58:57.475002 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:57.474959    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr" podStartSLOduration=32.929289359 podStartE2EDuration="34.47494533s" totalImagesPullingTime="1.545655971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:58:57.473901999 +0000 UTC m=+121.521326748" watchObservedRunningTime="2026-07-20 09:58:57.47494533 +0000 UTC m=+121.522370075"
Jul 20 09:58:58.424137 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:58.424105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:58.426393 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:58.426363    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94a1b7fa-4991-419f-9bdd-6f474c315059-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-npw5t\" (UID: \"94a1b7fa-4991-419f-9bdd-6f474c315059\") " pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:58.704369 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:58.704295    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wds68\""
Jul 20 09:58:58.711967 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:58.711951    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-npw5t"
Jul 20 09:58:58.872239 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:58.872209    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-npw5t"]
Jul 20 09:58:58.877852 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:58:58.877828    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a1b7fa_4991_419f_9bdd_6f474c315059.slice/crio-1741d0e42d008c36dd10c451f2ab77eaa91716a9c7d4c2311a0a3948f0203b44 WatchSource:0}: Error finding container 1741d0e42d008c36dd10c451f2ab77eaa91716a9c7d4c2311a0a3948f0203b44: Status 404 returned error can't find the container with id 1741d0e42d008c36dd10c451f2ab77eaa91716a9c7d4c2311a0a3948f0203b44
Jul 20 09:58:59.415043 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:59.415003    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:58:59.466379 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:58:59.466326    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npw5t" event={"ID":"94a1b7fa-4991-419f-9bdd-6f474c315059","Type":"ContainerStarted","Data":"1741d0e42d008c36dd10c451f2ab77eaa91716a9c7d4c2311a0a3948f0203b44"}
Jul 20 09:59:00.489711 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:00.489669    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:00.563288 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:00.563256    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:02.815728 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.815696    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"]
Jul 20 09:59:02.855023 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.854989    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674bb78d6b-pr868"]
Jul 20 09:59:02.855219 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.855125    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.857913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.857889    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9x5fc\""
Jul 20 09:59:02.857913 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.857890    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 09:59:02.858847 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.858830    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 09:59:02.858943 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.858856    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 20 09:59:02.858943 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.858857    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 09:59:02.864789 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.864765    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"]
Jul 20 09:59:02.864789 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.864788    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674bb78d6b-pr868"]
Jul 20 09:59:02.864981 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.864852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.867499 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.867468    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 09:59:02.867714 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.867699    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 09:59:02.867783 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.867707    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 09:59:02.867936 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.867919    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9q88t\""
Jul 20 09:59:02.869212 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.869186    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzm9\" (UniqueName: \"kubernetes.io/projected/b7709e41-76ef-4c05-94ef-72dc45810001-kube-api-access-4jzm9\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.869322 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.869303    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7709e41-76ef-4c05-94ef-72dc45810001-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.875463 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.875435    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 09:59:02.931606 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.931585    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"]
Jul 20 09:59:02.934596 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.934576    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-ft9gm"]
Jul 20 09:59:02.934730 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.934709    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:02.937363 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.937346    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"]
Jul 20 09:59:02.937529 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.937509    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:02.937967 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.937948    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 20 09:59:02.940217 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.940196    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"]
Jul 20 09:59:02.940308 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.940278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:02.943436 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943419    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 09:59:02.943545 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943451    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"
Jul 20 09:59:02.943662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943645    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 20 09:59:02.943770 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943755    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 09:59:02.943877 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943838    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wrxxt\""
Jul 20 09:59:02.943938 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.943929    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 20 09:59:02.946664 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.946647    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rws6t\""
Jul 20 09:59:02.946883 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.946865    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bnzgl\""
Jul 20 09:59:02.946968 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.946908    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gvnnz\""
Jul 20 09:59:02.958495 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.958479    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"]
Jul 20 09:59:02.959786 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.959764    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"]
Jul 20 09:59:02.966407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.966337    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-ft9gm"]
Jul 20 09:59:02.967662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.967646    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"]
Jul 20 09:59:02.969734 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969715    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:02.969843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969754    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.969843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969780    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzm9\" (UniqueName: \"kubernetes.io/projected/b7709e41-76ef-4c05-94ef-72dc45810001-kube-api-access-4jzm9\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.969843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969832    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970013 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969854    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:02.970013 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969892    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970013 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969918    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970013 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969966    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swl52\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970013 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.969989    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970234 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5z6h\" (UniqueName: \"kubernetes.io/projected/66a2f67e-595b-4feb-8250-0c2ee8467e32-kube-api-access-h5z6h\") pod \"network-check-source-7bc5b6bf85-wkvxw\" (UID: \"66a2f67e-595b-4feb-8250-0c2ee8467e32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"
Jul 20 09:59:02.970234 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970093    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970234 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7709e41-76ef-4c05-94ef-72dc45810001-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.970234 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970221    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb6pn\" (UniqueName: \"kubernetes.io/projected/268299b4-1d04-4c04-a83b-5557ffc339df-kube-api-access-cb6pn\") pod \"downloads-5bc67dcd46-ft9gm\" (UID: \"268299b4-1d04-4c04-a83b-5557ffc339df\") " pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:02.970375 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970244    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:02.970375 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970263    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/846d321e-a71e-4543-9695-10de6cf5e01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-nxr76\" (UID: \"846d321e-a71e-4543-9695-10de6cf5e01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:02.970977 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.970920    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674bb78d6b-pr868"]
Jul 20 09:59:02.971150 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:02.971126    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-swl52 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-674bb78d6b-pr868" podUID="cd6fbb67-351f-4e58-9c46-777f7a4890ba"
Jul 20 09:59:02.973014 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.972984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b7709e41-76ef-4c05-94ef-72dc45810001-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:02.991673 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:02.991613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzm9\" (UniqueName: \"kubernetes.io/projected/b7709e41-76ef-4c05-94ef-72dc45810001-kube-api-access-4jzm9\") pod \"managed-serviceaccount-addon-agent-5679c5f654-zgkrg\" (UID: \"b7709e41-76ef-4c05-94ef-72dc45810001\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:03.071383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071295    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.071383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071337    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:03.071383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.071383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071379    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.071704 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swl52\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.071704 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.071445    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.072244 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072216    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:03.072410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072298    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5z6h\" (UniqueName: \"kubernetes.io/projected/66a2f67e-595b-4feb-8250-0c2ee8467e32-kube-api-access-h5z6h\") pod \"network-check-source-7bc5b6bf85-wkvxw\" (UID: \"66a2f67e-595b-4feb-8250-0c2ee8467e32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"
Jul 20 09:59:03.072410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072327    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.072410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072386    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb6pn\" (UniqueName: \"kubernetes.io/projected/268299b4-1d04-4c04-a83b-5557ffc339df-kube-api-access-cb6pn\") pod \"downloads-5bc67dcd46-ft9gm\" (UID: \"268299b4-1d04-4c04-a83b-5557ffc339df\") " pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/846d321e-a71e-4543-9695-10de6cf5e01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-nxr76\" (UID: \"846d321e-a71e-4543-9695-10de6cf5e01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072483    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072512    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072696    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.073280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.072886    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.073601 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.073373    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.074398 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.074368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.074488 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.074420    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.074753 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.074717    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.075323 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.075298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.076052 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.076031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35ffa5a5-c4eb-4dfa-b27b-af607712fe9a-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-5s7hz\" (UID: \"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:03.076342 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.076324    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/846d321e-a71e-4543-9695-10de6cf5e01e-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-nxr76\" (UID: \"846d321e-a71e-4543-9695-10de6cf5e01e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:03.084158 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.084118    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.084843 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.084814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swl52\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52\") pod \"image-registry-674bb78d6b-pr868\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") " pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.085875 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.085850    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb6pn\" (UniqueName: \"kubernetes.io/projected/268299b4-1d04-4c04-a83b-5557ffc339df-kube-api-access-cb6pn\") pod \"downloads-5bc67dcd46-ft9gm\" (UID: \"268299b4-1d04-4c04-a83b-5557ffc339df\") " pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:03.087117 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.087049    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5z6h\" (UniqueName: \"kubernetes.io/projected/66a2f67e-595b-4feb-8250-0c2ee8467e32-kube-api-access-h5z6h\") pod \"network-check-source-7bc5b6bf85-wkvxw\" (UID: \"66a2f67e-595b-4feb-8250-0c2ee8467e32\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"
Jul 20 09:59:03.174615 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.174590    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"
Jul 20 09:59:03.210955 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.210924    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:03.244512 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.244491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:03.251936 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.251916    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:03.257914 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.257898    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"
Jul 20 09:59:03.264239 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.264220    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"
Jul 20 09:59:03.479772 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.479740    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.486956 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.486934    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:03.577719 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.577689    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.577896 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.577728    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.577896 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.577766    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578041 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578024    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swl52\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578090    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578122    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578198 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578150    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:03.578346 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578205    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578346 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578230    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets\") pod \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\" (UID: \"cd6fbb67-351f-4e58-9c46-777f7a4890ba\") "
Jul 20 09:59:03.578437 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578347    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 09:59:03.578518 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578498    2546 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-certificates\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.578588 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578525    2546 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd6fbb67-351f-4e58-9c46-777f7a4890ba-ca-trust-extracted\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.578932 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.578757    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:03.581027 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.580985    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52" (OuterVolumeSpecName: "kube-api-access-swl52") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "kube-api-access-swl52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:03.581027 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.581005    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:03.581204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.581043    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:03.581204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.581182    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:03.581307 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.581275    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cd6fbb67-351f-4e58-9c46-777f7a4890ba" (UID: "cd6fbb67-351f-4e58-9c46-777f7a4890ba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680637    2546 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-bound-sa-token\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680678    2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-registry-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680694    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swl52\" (UniqueName: \"kubernetes.io/projected/cd6fbb67-351f-4e58-9c46-777f7a4890ba-kube-api-access-swl52\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680710    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd6fbb67-351f-4e58-9c46-777f7a4890ba-trusted-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680726    2546 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-image-registry-private-configuration\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.681525 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.680742    2546 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd6fbb67-351f-4e58-9c46-777f7a4890ba-installation-pull-secrets\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:03.859012 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:03.858990    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"]
Jul 20 09:59:03.861108 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:03.861083    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846d321e_a71e_4543_9695_10de6cf5e01e.slice/crio-b87bb53c069a77ac50d79751259dfa2dbcc0267970f5f303e0713f9fb03cc5cf WatchSource:0}: Error finding container b87bb53c069a77ac50d79751259dfa2dbcc0267970f5f303e0713f9fb03cc5cf: Status 404 returned error can't find the container with id b87bb53c069a77ac50d79751259dfa2dbcc0267970f5f303e0713f9fb03cc5cf
Jul 20 09:59:04.012967 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.012943    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-ft9gm"]
Jul 20 09:59:04.014843 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:04.014810    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268299b4_1d04_4c04_a83b_5557ffc339df.slice/crio-3074abcc59cce6cbcfecb886fd2e4a93865ffad111c2c1fb4d52f70ace496e73 WatchSource:0}: Error finding container 3074abcc59cce6cbcfecb886fd2e4a93865ffad111c2c1fb4d52f70ace496e73: Status 404 returned error can't find the container with id 3074abcc59cce6cbcfecb886fd2e4a93865ffad111c2c1fb4d52f70ace496e73
Jul 20 09:59:04.025747 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.025703    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw"]
Jul 20 09:59:04.027367 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.027307    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz"]
Jul 20 09:59:04.028110 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:04.028082    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7709e41_76ef_4c05_94ef_72dc45810001.slice/crio-f1fe02b1d1260b900925c0c74c1644276c5d8b1124ce04d708bac43ffacaa700 WatchSource:0}: Error finding container f1fe02b1d1260b900925c0c74c1644276c5d8b1124ce04d708bac43ffacaa700: Status 404 returned error can't find the container with id f1fe02b1d1260b900925c0c74c1644276c5d8b1124ce04d708bac43ffacaa700
Jul 20 09:59:04.028648 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.028626    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg"]
Jul 20 09:59:04.095242 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.095217    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.190284 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.190249    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:04.485876 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.485787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npw5t" event={"ID":"94a1b7fa-4991-419f-9bdd-6f474c315059","Type":"ContainerStarted","Data":"9d98db0e65c0d300f1280988d701a320258e39a99a9b8f62e4f2ea5a0de19540"}
Jul 20 09:59:04.487485 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.487458    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76" event={"ID":"846d321e-a71e-4543-9695-10de6cf5e01e","Type":"ContainerStarted","Data":"b87bb53c069a77ac50d79751259dfa2dbcc0267970f5f303e0713f9fb03cc5cf"}
Jul 20 09:59:04.489011 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.488985    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz" event={"ID":"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a","Type":"ContainerStarted","Data":"f59fdad1f6adbd42a17be34ba3c6536d00e32f6d03ae1cbf30f879143fa5669e"}
Jul 20 09:59:04.491857 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.491415    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw" event={"ID":"66a2f67e-595b-4feb-8250-0c2ee8467e32","Type":"ContainerStarted","Data":"bd3627acfe286cdba1df70b624955533e85c7a56fe18a7d0cc08207bb0aaf81e"}
Jul 20 09:59:04.491857 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.491466    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw" event={"ID":"66a2f67e-595b-4feb-8250-0c2ee8467e32","Type":"ContainerStarted","Data":"4c6ae75665e7404358740ad3bd4d44808c1790fc0b56133aa7d39335b408d266"}
Jul 20 09:59:04.494353 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.494321    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-ft9gm" event={"ID":"268299b4-1d04-4c04-a83b-5557ffc339df","Type":"ContainerStarted","Data":"3074abcc59cce6cbcfecb886fd2e4a93865ffad111c2c1fb4d52f70ace496e73"}
Jul 20 09:59:04.495893 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.495660    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674bb78d6b-pr868"
Jul 20 09:59:04.495893 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.495871    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg" event={"ID":"b7709e41-76ef-4c05-94ef-72dc45810001","Type":"ContainerStarted","Data":"f1fe02b1d1260b900925c0c74c1644276c5d8b1124ce04d708bac43ffacaa700"}
Jul 20 09:59:04.512479 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.511513    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-wkvxw" podStartSLOduration=2.511496023 podStartE2EDuration="2.511496023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:04.51018103 +0000 UTC m=+128.557605779" watchObservedRunningTime="2026-07-20 09:59:04.511496023 +0000 UTC m=+128.558920769"
Jul 20 09:59:04.551384 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.551316    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674bb78d6b-pr868"]
Jul 20 09:59:04.552694 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:04.552642    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-674bb78d6b-pr868"]
Jul 20 09:59:06.309846 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.309803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:59:06.312982 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.312927    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44618c9c-a6c5-4e94-aedf-cadacba06315-metrics-certs\") pod \"network-metrics-daemon-5qzr2\" (UID: \"44618c9c-a6c5-4e94-aedf-cadacba06315\") " pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:59:06.404754 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.404714    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6fbb67-351f-4e58-9c46-777f7a4890ba" path="/var/lib/kubelet/pods/cd6fbb67-351f-4e58-9c46-777f7a4890ba/volumes"
Jul 20 09:59:06.614968 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.614939    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b7v24\""
Jul 20 09:59:06.620436 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.620408    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5qzr2"
Jul 20 09:59:06.651729 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.651628    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:06.722434 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:06.722403    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.013468 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.013432    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.109705 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.109589    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.202937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.202909    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5qzr2"]
Jul 20 09:59:10.207322 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:10.207293    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44618c9c_a6c5_4e94_aedf_cadacba06315.slice/crio-92c6c0a3dcdad342eadc494d2067cb54939980026d939c34b6890fd1362c587c WatchSource:0}: Error finding container 92c6c0a3dcdad342eadc494d2067cb54939980026d939c34b6890fd1362c587c: Status 404 returned error can't find the container with id 92c6c0a3dcdad342eadc494d2067cb54939980026d939c34b6890fd1362c587c
Jul 20 09:59:10.536181 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.536098    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg" event={"ID":"b7709e41-76ef-4c05-94ef-72dc45810001","Type":"ContainerStarted","Data":"21d29fbdfff6d7d8557601f1795f25918348be2a0b5ca330a302e238aed76021"}
Jul 20 09:59:10.538115 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.538081    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npw5t" event={"ID":"94a1b7fa-4991-419f-9bdd-6f474c315059","Type":"ContainerStarted","Data":"56f16404de92f3e10fb62add18d73dfb544a67386c64e9c4fc8f5813c1d221f6"}
Jul 20 09:59:10.539696 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.539670    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76" event={"ID":"846d321e-a71e-4543-9695-10de6cf5e01e","Type":"ContainerStarted","Data":"1dc2b5d3e92ec024c799bfe66b00d45b858def40dd8b4a7a19ae9c92d3254d28"}
Jul 20 09:59:10.539899 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.539849    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:10.541456 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.541392    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz" event={"ID":"35ffa5a5-c4eb-4dfa-b27b-af607712fe9a","Type":"ContainerStarted","Data":"5d64ce5978c479f6cc86180176c252898da102ac391e1dac730da3567c16d391"}
Jul 20 09:59:10.542711 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.542686    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qzr2" event={"ID":"44618c9c-a6c5-4e94-aedf-cadacba06315","Type":"ContainerStarted","Data":"92c6c0a3dcdad342eadc494d2067cb54939980026d939c34b6890fd1362c587c"}
Jul 20 09:59:10.545334 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.545302    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76"
Jul 20 09:59:10.558838 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.558785    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5679c5f654-zgkrg" podStartSLOduration=2.532068291 podStartE2EDuration="8.558771053s" totalImagesPullingTime="6.026702762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:10.557802303 +0000 UTC m=+134.605227049" watchObservedRunningTime="2026-07-20 09:59:10.558771053 +0000 UTC m=+134.606195799"
Jul 20 09:59:10.577166 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.577118    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-nxr76" podStartSLOduration=2.26116628 podStartE2EDuration="8.577101669s" totalImagesPullingTime="6.315935389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:10.57536662 +0000 UTC m=+134.622791367" watchObservedRunningTime="2026-07-20 09:59:10.577101669 +0000 UTC m=+134.624526394"
Jul 20 09:59:10.596809 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.596563    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-5s7hz" podStartSLOduration=3.229543639 podStartE2EDuration="8.596546582s" totalImagesPullingTime="5.367002943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:10.595244386 +0000 UTC m=+134.642669132" watchObservedRunningTime="2026-07-20 09:59:10.596546582 +0000 UTC m=+134.643971328"
Jul 20 09:59:10.713818 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.713776    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.821912 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.821805    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:10.918004 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:10.917890    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.154216 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.154164    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.299744 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.299694    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.398534 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.398474    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.550688 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.550492    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qzr2" event={"ID":"44618c9c-a6c5-4e94-aedf-cadacba06315","Type":"ContainerStarted","Data":"c810277999eef73caf33f1257e20fd7d04c9a6facdcff2c6d6cc8a1da6fac960"}
Jul 20 09:59:11.550688 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.550553    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5qzr2" event={"ID":"44618c9c-a6c5-4e94-aedf-cadacba06315","Type":"ContainerStarted","Data":"96dd9ca112a2056623e054de3662c953bfc25dfa747cf14cc56adcbffdc827d4"}
Jul 20 09:59:11.573974 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.573829    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5qzr2" podStartSLOduration=134.550596408 podStartE2EDuration="2m15.573810384s" totalImagesPullingTime="1.023213976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:11.573419639 +0000 UTC m=+135.620844385" watchObservedRunningTime="2026-07-20 09:59:11.573810384 +0000 UTC m=+135.621235130"
Jul 20 09:59:11.696505 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.696435    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.800422 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.800030    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:11.896908 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:11.896873    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:12.042841 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:12.042721    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:13.876585 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:13.876490    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:13.979368 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:13.979333    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:14.565491 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:14.565407    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-npw5t" event={"ID":"94a1b7fa-4991-419f-9bdd-6f474c315059","Type":"ContainerStarted","Data":"f5cf7ccf9010566d9530a5b212612d26e9a3e802843e32dfa09a418ca6611855"}
Jul 20 09:59:14.593306 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:14.593250    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-npw5t" podStartSLOduration=33.349174704 podStartE2EDuration="48.593233753s" totalImagesPullingTime="15.244059049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 09:58:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:14.592721979 +0000 UTC m=+138.640146725" watchObservedRunningTime="2026-07-20 09:59:14.593233753 +0000 UTC m=+138.640658497"
Jul 20 09:59:15.034205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.034173    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:15.038409 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.038381    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.047396 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.047372    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 09:59:15.049479 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.049456    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 09:59:15.049600 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.049468    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 09:59:15.049995 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.049975    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7mwdt\""
Jul 20 09:59:15.050100 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.049994    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 09:59:15.050366 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.050345    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 09:59:15.057278 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.057252    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 09:59:15.060569 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.060546    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:15.085012 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.084982    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085127    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085286 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085168    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085286 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.085286 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.085265    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88mm\" (UniqueName: \"kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186340 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186311    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186463 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186354    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186463 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186415    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88mm\" (UniqueName: \"kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186581 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186488    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186581 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186581 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186557    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.186676 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.186582    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.187226 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.187202    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.187365 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.187226    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.187539 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.187489    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.187656 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.187609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.189055 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.189030    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.189523 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.189501    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.196024 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.196003    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88mm\" (UniqueName: \"kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm\") pod \"console-76f66965bf-qmdng\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") " pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.349729 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.349691    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:15.397923 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.397891    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:15.639910 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.639794    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:15.698126 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:15.698059    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.345327 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.345276    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.468975 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.468921    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.649430 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.649338    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.825670 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.825633    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6xqsv"]
Jul 20 09:59:17.829049 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.829026    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.832522 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.832500    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sw4tj\""
Jul 20 09:59:17.833115 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.832956    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 09:59:17.833115 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.832984    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 09:59:17.833115 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.832993    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 09:59:17.833863 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.833843    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 09:59:17.855329 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.855277    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:17.907007 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.906933    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907007 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.906976    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-wtmp\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907007 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907003    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907249 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907041    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-root\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907249 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907249 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907149    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwt52\" (UniqueName: \"kubernetes.io/projected/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-kube-api-access-qwt52\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907249 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907227    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-textfile\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907251    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-sys\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.907407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.907284    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-metrics-client-ca\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:17.985759 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:17.985721    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.008320 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008294    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-textfile\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008331    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-sys\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008358    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-metrics-client-ca\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008425    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-sys\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008431    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008460 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008454    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-wtmp\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:18.008536    2546 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008577    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-wtmp\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:18.008598    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls podName:2e62c52e-f97c-4dd0-8885-8a8aa90eb536 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:18.508579831 +0000 UTC m=+142.556004575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls") pod "node-exporter-6xqsv" (UID: "2e62c52e-f97c-4dd0-8885-8a8aa90eb536") : secret "node-exporter-tls" not found
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008599    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008647    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-root\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.008723 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008681    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.009017 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008736    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-root\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.009017 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008713    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwt52\" (UniqueName: \"kubernetes.io/projected/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-kube-api-access-qwt52\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.009017 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008680    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-textfile\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.009017 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.008981    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-metrics-client-ca\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.009237 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.009118    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.011225 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.011199    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.022316 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.022294    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwt52\" (UniqueName: \"kubernetes.io/projected/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-kube-api-access-qwt52\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.099628 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.099594    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.226720 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.226635    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.320353 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.320316    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.421530 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.421499    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.513127 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.513034    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.515576 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.515553    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2e62c52e-f97c-4dd0-8885-8a8aa90eb536-node-exporter-tls\") pod \"node-exporter-6xqsv\" (UID: \"2e62c52e-f97c-4dd0-8885-8a8aa90eb536\") " pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.522216 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.522185    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.634247 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.634211    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.733523 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.733487    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.740407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.740378    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xqsv"
Jul 20 09:59:18.759247 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.757847    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:18.762916 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.762893    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.767033 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.766954    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 20 09:59:18.767205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767135    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 20 09:59:18.767281 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767270    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 20 09:59:18.767337 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767277    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 20 09:59:18.767415 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767389    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 20 09:59:18.767707 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767689    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 20 09:59:18.767962 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.767945    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 20 09:59:18.768176 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.768161    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 20 09:59:18.769567 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.769549    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qp8dt\""
Jul 20 09:59:18.771126 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.770966    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 20 09:59:18.781686 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.781664    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:18.816443 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816412    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816452    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816483    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816542    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816582    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816615    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816662    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816720    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kb8j\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816744    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816771    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.816822 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816813    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.817138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816858    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.817138 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.816938    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.861372 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.861341    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:18.917896 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.917864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.917896 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.917897    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.917926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.917956    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.917983    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918010    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918102    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kb8j\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918132 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918485 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918157    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918485 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918206    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918485 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918234    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.918485 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.918275    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.919903 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.919871    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.920360 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.920163    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.920360 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:18.920292    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle podName:29a16b8a-5447-4c09-b4a8-f879ebf40504 nodeName:}" failed. No retries permitted until 2026-07-20 09:59:19.4202718 +0000 UTC m=+143.467696525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504") : configmap references non-existent config key: ca-bundle.crt
Jul 20 09:59:18.921861 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.921838    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.923515 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.923347    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.923515 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.923486    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.923662 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.923618    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.923937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.923916    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.924045 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.924028    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.924286 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.924256    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.924433 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.924409    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.924974 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.924949    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:18.928798 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:18.928780    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kb8j\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:19.422660 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:19.422626    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:19.423471 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:19.423441    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:19.673753 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:19.673686    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 09:59:20.623630 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.623590    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76cf56f478-zqrdd"]
Jul 20 09:59:20.657469 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.657444    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76cf56f478-zqrdd"]
Jul 20 09:59:20.657639 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.657583    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.661435 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.661404    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 20 09:59:20.661435 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.661428    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 20 09:59:20.661696 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.661679    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 20 09:59:20.662046 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.662020    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 20 09:59:20.663229 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.663108    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tjnwb\""
Jul 20 09:59:20.663229 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.663121    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 20 09:59:20.663229 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.663108    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eg1isibfalqj9\""
Jul 20 09:59:20.735998 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.735966    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-grpc-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736182 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736008    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736182 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnkz\" (UniqueName: \"kubernetes.io/projected/58098f4e-dd1a-480b-8c8d-48a83c40eb00-kube-api-access-zwnkz\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736182 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736173    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736356 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736211    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736356 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58098f4e-dd1a-480b-8c8d-48a83c40eb00-metrics-client-ca\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736459 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736396    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.736459 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.736448    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.833985 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.833949    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:20.837891 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.837866    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.837913    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.837953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-grpc-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.837987    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.838038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnkz\" (UniqueName: \"kubernetes.io/projected/58098f4e-dd1a-480b-8c8d-48a83c40eb00-kube-api-access-zwnkz\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.838102    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.838144    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.838204 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.838169    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58098f4e-dd1a-480b-8c8d-48a83c40eb00-metrics-client-ca\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.839030 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.838972    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58098f4e-dd1a-480b-8c8d-48a83c40eb00-metrics-client-ca\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841259 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841229    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-grpc-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841664 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841638    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841664 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841658    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-tls\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841818 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841691    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841818 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841696    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.841917 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.841873    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58098f4e-dd1a-480b-8c8d-48a83c40eb00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.847475 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.847450    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnkz\" (UniqueName: \"kubernetes.io/projected/58098f4e-dd1a-480b-8c8d-48a83c40eb00-kube-api-access-zwnkz\") pod \"thanos-querier-76cf56f478-zqrdd\" (UID: \"58098f4e-dd1a-480b-8c8d-48a83c40eb00\") " pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:20.930335 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.930245    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:20.969933 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:20.969903    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:21.203360 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:21.203276    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:21.311530 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:21.311493    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:22.164970 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:22.164929    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e62c52e_f97c_4dd0_8885_8a8aa90eb536.slice/crio-42b5795143f81721662aeb3650e7a4a1dac6848939e7923a43d73fbf5141c975 WatchSource:0}: Error finding container 42b5795143f81721662aeb3650e7a4a1dac6848939e7923a43d73fbf5141c975: Status 404 returned error can't find the container with id 42b5795143f81721662aeb3650e7a4a1dac6848939e7923a43d73fbf5141c975
Jul 20 09:59:22.238526 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.238394    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75b788647d-bb5bd"]
Jul 20 09:59:22.248446 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.248420    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.251973 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.251535    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gq5fv\""
Jul 20 09:59:22.251973 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.251809    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 20 09:59:22.254117 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.252588    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-26tin88no0eo6\""
Jul 20 09:59:22.254117 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.252868    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 20 09:59:22.254117 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.254047    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75b788647d-bb5bd"]
Jul 20 09:59:22.254350 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.254251    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 20 09:59:22.254937 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.254502    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 20 09:59:22.326359 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.326325    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:22.352546 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352519    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-client-certs\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352645 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352562    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-client-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352645 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352612    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4vz4\" (UniqueName: \"kubernetes.io/projected/ba6b2010-37a4-4601-9589-b5710893e6de-kube-api-access-t4vz4\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352645 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-metrics-server-audit-profiles\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352771 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba6b2010-37a4-4601-9589-b5710893e6de-audit-log\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352771 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352723    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-tls\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.352771 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.352758    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.388136 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:22.388062    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8738652_cc9a_4df3_981c_4c329a13baff.slice/crio-896c900adcb7d71d17a90519d0d1b6cd89fe36ab9747a0d1f39bf92568faaa11 WatchSource:0}: Error finding container 896c900adcb7d71d17a90519d0d1b6cd89fe36ab9747a0d1f39bf92568faaa11: Status 404 returned error can't find the container with id 896c900adcb7d71d17a90519d0d1b6cd89fe36ab9747a0d1f39bf92568faaa11
Jul 20 09:59:22.419310 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.419252    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:22.453400 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453367    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-client-certs\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453403    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-client-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453434    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4vz4\" (UniqueName: \"kubernetes.io/projected/ba6b2010-37a4-4601-9589-b5710893e6de-kube-api-access-t4vz4\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-metrics-server-audit-profiles\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453537 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba6b2010-37a4-4601-9589-b5710893e6de-audit-log\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453726 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453556    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-tls\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.453726 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.453605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.454036 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.454009    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ba6b2010-37a4-4601-9589-b5710893e6de-audit-log\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.454368 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.454349    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.454655 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.454633    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ba6b2010-37a4-4601-9589-b5710893e6de-metrics-server-audit-profiles\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.456037 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.456018    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-client-ca-bundle\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.456508 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.456488    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-client-certs\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.456555 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.456516    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ba6b2010-37a4-4601-9589-b5710893e6de-secret-metrics-server-tls\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.470683 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.470665    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4vz4\" (UniqueName: \"kubernetes.io/projected/ba6b2010-37a4-4601-9589-b5710893e6de-kube-api-access-t4vz4\") pod \"metrics-server-75b788647d-bb5bd\" (UID: \"ba6b2010-37a4-4601-9589-b5710893e6de\") " pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.528215 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.528192    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:22.547834 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.547812    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76cf56f478-zqrdd"]
Jul 20 09:59:22.549219 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.549197    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 09:59:22.549825 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:22.549799    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58098f4e_dd1a_480b_8c8d_48a83c40eb00.slice/crio-dc8d1db4f9f96c707270f3707c88fb1e2ceeca7114a129b8963e4bdb705cc667 WatchSource:0}: Error finding container dc8d1db4f9f96c707270f3707c88fb1e2ceeca7114a129b8963e4bdb705cc667: Status 404 returned error can't find the container with id dc8d1db4f9f96c707270f3707c88fb1e2ceeca7114a129b8963e4bdb705cc667
Jul 20 09:59:22.550522 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:22.550500    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a16b8a_5447_4c09_b4a8_f879ebf40504.slice/crio-ff18df72df8236208e127927b325c2a3c1db70b96aaaff77eb9b9aeda6868787 WatchSource:0}: Error finding container ff18df72df8236208e127927b325c2a3c1db70b96aaaff77eb9b9aeda6868787: Status 404 returned error can't find the container with id ff18df72df8236208e127927b325c2a3c1db70b96aaaff77eb9b9aeda6868787
Jul 20 09:59:22.564952 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.564935    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:22.599637 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.599605    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f66965bf-qmdng" event={"ID":"e8738652-cc9a-4df3-981c-4c329a13baff","Type":"ContainerStarted","Data":"896c900adcb7d71d17a90519d0d1b6cd89fe36ab9747a0d1f39bf92568faaa11"}
Jul 20 09:59:22.600772 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.600750    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xqsv" event={"ID":"2e62c52e-f97c-4dd0-8885-8a8aa90eb536","Type":"ContainerStarted","Data":"42b5795143f81721662aeb3650e7a4a1dac6848939e7923a43d73fbf5141c975"}
Jul 20 09:59:22.601942 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.601918    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"ff18df72df8236208e127927b325c2a3c1db70b96aaaff77eb9b9aeda6868787"}
Jul 20 09:59:22.603254 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.603232    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"dc8d1db4f9f96c707270f3707c88fb1e2ceeca7114a129b8963e4bdb705cc667"}
Jul 20 09:59:22.604790 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.604765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-ft9gm" event={"ID":"268299b4-1d04-4c04-a83b-5557ffc339df","Type":"ContainerStarted","Data":"16471c9a4cc729708f846bdee6d256e877dce761f276c54803ce403598f5d5ae"}
Jul 20 09:59:22.605015 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.604994    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:22.606410 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.606378    2546 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-ft9gm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused" start-of-body=
Jul 20 09:59:22.606508 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.606420    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-ft9gm" podUID="268299b4-1d04-4c04-a83b-5557ffc339df" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.19:8080/\": dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 20 09:59:22.648722 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.648694    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:22.725217 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.725169    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-ft9gm" podStartSLOduration=2.266220485 podStartE2EDuration="20.725149117s" totalImagesPullingTime="18.458928632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:22.635731121 +0000 UTC m=+146.683155867" watchObservedRunningTime="2026-07-20 09:59:22.725149117 +0000 UTC m=+146.772573857"
Jul 20 09:59:22.725981 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:22.725961    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75b788647d-bb5bd"]
Jul 20 09:59:22.728337 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:22.728313    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6b2010_37a4_4601_9589_b5710893e6de.slice/crio-38a5775372114f50618fc85a73485a227d298d9592c1ed9166d47ffeed1c4c02 WatchSource:0}: Error finding container 38a5775372114f50618fc85a73485a227d298d9592c1ed9166d47ffeed1c4c02: Status 404 returned error can't find the container with id 38a5775372114f50618fc85a73485a227d298d9592c1ed9166d47ffeed1c4c02
Jul 20 09:59:23.027019 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.026630    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.104025 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.103300    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8467c64b86-hfwvj"]
Jul 20 09:59:23.120092 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.119663    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.133176    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.133464    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.133640    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.134254    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.134465    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 20 09:59:23.134918 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.134643    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bnsrd\""
Jul 20 09:59:23.137611 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.137535    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8467c64b86-hfwvj"]
Jul 20 09:59:23.143167 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.142410    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 20 09:59:23.144261 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.144126    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:23.261322 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261267    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261401    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-federate-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261460    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261489    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-metrics-client-ca\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261558    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-serving-certs-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261588    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261612    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59qf\" (UniqueName: \"kubernetes.io/projected/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-kube-api-access-f59qf\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.261835 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.261641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.268216 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.268139    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:23.306147 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.305443    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 09:59:23.342077 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.342023    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 09:59:23.342285 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.342180    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363281    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-federate-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363338    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363369    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-metrics-client-ca\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363423    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-serving-certs-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363452    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363476    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59qf\" (UniqueName: \"kubernetes.io/projected/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-kube-api-access-f59qf\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363503    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.363727 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.363557    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.366850 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.364982    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.366850 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.366370    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-serving-certs-ca-bundle\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.384217 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.375433    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-metrics-client-ca\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.384217 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.377382    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.388551 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.388520    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59qf\" (UniqueName: \"kubernetes.io/projected/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-kube-api-access-f59qf\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.388682 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.388663    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-telemeter-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.390868 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.390841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-federate-client-tls\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.392445 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.392405    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef-secret-telemeter-client\") pod \"telemeter-client-8467c64b86-hfwvj\" (UID: \"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef\") " pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.445893 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.445676    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464434    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464494    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464635    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464667    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.464892 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.464741    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxcd\" (UniqueName: \"kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.566642 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.566819 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxcd\" (UniqueName: \"kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.566819 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566703    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.566819 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566745    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.567205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566819    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.567205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566858    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.567205 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.566890    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.568160 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.568131    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.568487 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.568406    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.568608 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.568485    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.569565 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.569352    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.582438 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.582388    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.585449 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.585389    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxcd\" (UniqueName: \"kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.587771 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.587745    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert\") pod \"console-74fbc9ff54-zs7x4\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") " pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:23.615221 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.615118    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd" event={"ID":"ba6b2010-37a4-4601-9589-b5710893e6de","Type":"ContainerStarted","Data":"38a5775372114f50618fc85a73485a227d298d9592c1ed9166d47ffeed1c4c02"}
Jul 20 09:59:23.619704 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.619441    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-ft9gm"
Jul 20 09:59:23.658185 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:23.655468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:24.195607 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:24.195437    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:26.432978 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:26.432896    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:26.554275 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:26.554184    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:27.349614 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:27.349569    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:27.447881 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:27.447750    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:28.345604 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.345569    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8467c64b86-hfwvj"]
Jul 20 09:59:28.353552 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:28.351540    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2adc5a_9e60_4ce0_a0c8_fd63f0f2d5ef.slice/crio-526e2c45819ca59047aaace514bfaf27ffee1ec7e1560da9e4d16dc392b7bc21 WatchSource:0}: Error finding container 526e2c45819ca59047aaace514bfaf27ffee1ec7e1560da9e4d16dc392b7bc21: Status 404 returned error can't find the container with id 526e2c45819ca59047aaace514bfaf27ffee1ec7e1560da9e4d16dc392b7bc21
Jul 20 09:59:28.484280 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.484254    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 09:59:28.487230 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:28.487152    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc1968_b4cc_465a_b0f7_12871dadbee6.slice/crio-4429e5cff57bb68a55f43182dfc92529126758515a2ec1f13f314ca1adc52734 WatchSource:0}: Error finding container 4429e5cff57bb68a55f43182dfc92529126758515a2ec1f13f314ca1adc52734: Status 404 returned error can't find the container with id 4429e5cff57bb68a55f43182dfc92529126758515a2ec1f13f314ca1adc52734
Jul 20 09:59:28.649425 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.648740    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e62c52e-f97c-4dd0-8885-8a8aa90eb536" containerID="441d7ff89c58b92db960a88697e6fe9384070d0f935867968912a7d24591bfa4" exitCode=0
Jul 20 09:59:28.649425 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.648847    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xqsv" event={"ID":"2e62c52e-f97c-4dd0-8885-8a8aa90eb536","Type":"ContainerDied","Data":"441d7ff89c58b92db960a88697e6fe9384070d0f935867968912a7d24591bfa4"}
Jul 20 09:59:28.652147 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.652120    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a" exitCode=0
Jul 20 09:59:28.652236 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.652218    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a"}
Jul 20 09:59:28.656951 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.656810    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"98d93a8bd00f060cf78257356a292fe9e3d24df87b29e9e340c0f1891f30372f"}
Jul 20 09:59:28.656951 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.656878    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"d17b4cce2ab507d5ccd0d87f227f08c634fcf9355ed2a9c165be876077901f45"}
Jul 20 09:59:28.660403 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.660379    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc9ff54-zs7x4" event={"ID":"8adc1968-b4cc-465a-b0f7-12871dadbee6","Type":"ContainerStarted","Data":"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"}
Jul 20 09:59:28.661220 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.661159    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc9ff54-zs7x4" event={"ID":"8adc1968-b4cc-465a-b0f7-12871dadbee6","Type":"ContainerStarted","Data":"4429e5cff57bb68a55f43182dfc92529126758515a2ec1f13f314ca1adc52734"}
Jul 20 09:59:28.664887 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.664857    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f66965bf-qmdng" event={"ID":"e8738652-cc9a-4df3-981c-4c329a13baff","Type":"ContainerStarted","Data":"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"}
Jul 20 09:59:28.666517 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.666497    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd" event={"ID":"ba6b2010-37a4-4601-9589-b5710893e6de","Type":"ContainerStarted","Data":"1d21529e85389ec9ddd59d61c17b0982c340fa33d665de98c0d753ed71e808e3"}
Jul 20 09:59:28.669340 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.669318    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj" event={"ID":"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef","Type":"ContainerStarted","Data":"526e2c45819ca59047aaace514bfaf27ffee1ec7e1560da9e4d16dc392b7bc21"}
Jul 20 09:59:28.747199 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.747056    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fbc9ff54-zs7x4" podStartSLOduration=5.74703407 podStartE2EDuration="5.74703407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:28.721967021 +0000 UTC m=+152.769391768" watchObservedRunningTime="2026-07-20 09:59:28.74703407 +0000 UTC m=+152.794458818"
Jul 20 09:59:28.748736 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.748613    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd" podStartSLOduration=1.129603321 podStartE2EDuration="6.748600516s" totalImagesPullingTime="5.618997195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:28.7460413 +0000 UTC m=+152.793466047" watchObservedRunningTime="2026-07-20 09:59:28.748600516 +0000 UTC m=+152.796025262"
Jul 20 09:59:28.768110 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.767878    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76f66965bf-qmdng" podStartSLOduration=7.81213332 podStartE2EDuration="13.767861381s" totalImagesPullingTime="5.955728061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:28.764750162 +0000 UTC m=+152.812174910" watchObservedRunningTime="2026-07-20 09:59:28.767861381 +0000 UTC m=+152.815286127"
Jul 20 09:59:28.856032 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.855987    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:28.956307 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:28.956181    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:29.681812 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:29.681691    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xqsv" event={"ID":"2e62c52e-f97c-4dd0-8885-8a8aa90eb536","Type":"ContainerStarted","Data":"47882e52e2905f2a7123ec4daaf5f0db47a1bcb6b0a87924cf27451a6351029e"}
Jul 20 09:59:29.682447 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:29.682388    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xqsv" event={"ID":"2e62c52e-f97c-4dd0-8885-8a8aa90eb536","Type":"ContainerStarted","Data":"d0fbd45a16ea4e395eadde66eb685db0d5a0660ca8420172e5f1f149636255ec"}
Jul 20 09:59:29.688773 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:29.688413    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"4b1b2e7a2d2e180e0bbde78deeface9e0083e3cec68dbcfdaa52dcb924fcd418"}
Jul 20 09:59:29.705643 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:29.705109    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6xqsv" podStartSLOduration=6.569216015 podStartE2EDuration="12.705089201s" totalImagesPullingTime="6.020225178s" totalInitContainerRuntime="115.648008ms" isStatefulPod=true podCreationTimestamp="2026-07-20 09:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:29.703185157 +0000 UTC m=+153.750609904" watchObservedRunningTime="2026-07-20 09:59:29.705089201 +0000 UTC m=+153.752513942"
Jul 20 09:59:30.488897 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:30.488858    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:30.593116 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:30.593049    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:31.700830 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:31.700803    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"b1b219015f3996c63ff22c854181f251190a9af49424dc695a49ebd9985579c3"}
Jul 20 09:59:31.702365 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:31.702339    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj" event={"ID":"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef","Type":"ContainerStarted","Data":"089a697fbc106a86a420292c0238e1273750c87f4bc4df778e7cd98dd636f09e"}
Jul 20 09:59:31.704382 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:31.704325    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"}
Jul 20 09:59:32.712004 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.711957    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"}
Jul 20 09:59:32.712621 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.712040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"}
Jul 20 09:59:32.712621 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.712109    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"}
Jul 20 09:59:32.712621 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.712140    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"}
Jul 20 09:59:32.712621 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.712171    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerStarted","Data":"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"}
Jul 20 09:59:32.714994 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.714962    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"d8fc0c49941c6d1734c67519db72553edecce96650727c297487eb8574de2f60"}
Jul 20 09:59:32.715144 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.715032    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" event={"ID":"58098f4e-dd1a-480b-8c8d-48a83c40eb00","Type":"ContainerStarted","Data":"9ec30852cb3b2ae7cfaf79e836786c394e3877d21f960ffc492da8e11dc50ad5"}
Jul 20 09:59:32.715144 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.715124    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:32.716925 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.716902    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj" event={"ID":"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef","Type":"ContainerStarted","Data":"9adb5a9bbc693297b124eab457a22a9368e8434b3fe2ebd549eb9fe9a7f35a8b"}
Jul 20 09:59:32.717049 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.716950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj" event={"ID":"9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef","Type":"ContainerStarted","Data":"e36d1f1b2fedcfd88dd779a31739da6f76e271d4009fd5a9957949ca8d0e64ae"}
Jul 20 09:59:32.743142 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.743103    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.055332362 podStartE2EDuration="14.743090501s" totalImagesPullingTime="8.676088655s" totalInitContainerRuntime="11.669484ms" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:32.740594803 +0000 UTC m=+156.788019550" watchObservedRunningTime="2026-07-20 09:59:32.743090501 +0000 UTC m=+156.790515243"
Jul 20 09:59:32.777149 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.777096    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd" podStartSLOduration=4.186668492 podStartE2EDuration="12.777057662s" totalImagesPullingTime="8.59038917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:32.77525578 +0000 UTC m=+156.822680524" watchObservedRunningTime="2026-07-20 09:59:32.777057662 +0000 UTC m=+156.824482408"
Jul 20 09:59:32.820519 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:32.820471    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8467c64b86-hfwvj" podStartSLOduration=6.606289343 podStartE2EDuration="9.820457469s" totalImagesPullingTime="3.214168126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:32.819588802 +0000 UTC m=+156.867013571" watchObservedRunningTime="2026-07-20 09:59:32.820457469 +0000 UTC m=+156.867882213"
Jul 20 09:59:33.658953 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.658921    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:33.659283 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.659260    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:33.663576 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.663553    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:33.700818 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.700793    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 09:59:33.726477 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.726455    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 09:59:33.727738 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.727718    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76cf56f478-zqrdd"
Jul 20 09:59:33.742359 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.742331    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 09:59:33.761395 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.761371    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.766778 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.766759    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 09:59:33.876689 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.876656    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.876861 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.876771    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.876861 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.876841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.877053 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.877028    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.877414 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.877384    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.877572 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.877549    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zwfz\" (UniqueName: \"kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.877657 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.877643    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.978952    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zwfz\" (UniqueName: \"kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979016 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.978989    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979048    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979091    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979122    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979156    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979202 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979188    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979796 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979768    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979898 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979877    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979961 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979896    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.979961 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.979946    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.981631 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.981601    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.981722 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.981687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:33.987809 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:33.987783    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zwfz\" (UniqueName: \"kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz\") pod \"console-67c4f65cc-r9k2q\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") " pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:34.071422 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:34.071397    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:34.222658 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:34.222636    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 09:59:34.225043 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:34.225015    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7b9606_e139_404e_881f_2340d8005916.slice/crio-98d98fb70ed9a99fd65c4c7e09901a7faa8328cdfd1491db665f139b6ec489c9 WatchSource:0}: Error finding container 98d98fb70ed9a99fd65c4c7e09901a7faa8328cdfd1491db665f139b6ec489c9: Status 404 returned error can't find the container with id 98d98fb70ed9a99fd65c4c7e09901a7faa8328cdfd1491db665f139b6ec489c9
Jul 20 09:59:34.727459 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:34.727418    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c4f65cc-r9k2q" event={"ID":"0c7b9606-e139-404e-881f-2340d8005916","Type":"ContainerStarted","Data":"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"}
Jul 20 09:59:34.727857 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:34.727478    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c4f65cc-r9k2q" event={"ID":"0c7b9606-e139-404e-881f-2340d8005916","Type":"ContainerStarted","Data":"98d98fb70ed9a99fd65c4c7e09901a7faa8328cdfd1491db665f139b6ec489c9"}
Jul 20 09:59:34.749285 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:34.749241    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67c4f65cc-r9k2q" podStartSLOduration=1.749226467 podStartE2EDuration="1.749226467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:34.748163938 +0000 UTC m=+158.795588683" watchObservedRunningTime="2026-07-20 09:59:34.749226467 +0000 UTC m=+158.796651212"
Jul 20 09:59:34.928563 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:34.928524    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-94j7v" podUID="2eb3e353-a052-4603-917b-754e8fef71b2"
Jul 20 09:59:35.350077 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:35.350049    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:35.732277 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:35.732204    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:38.331139 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:38.331095    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.479767 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:38.479734    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:38.610828 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:38.610737    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:39.825323 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:39.825286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:39.827533 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:39.827515    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2eb3e353-a052-4603-917b-754e8fef71b2-metrics-tls\") pod \"dns-default-94j7v\" (UID: \"2eb3e353-a052-4603-917b-754e8fef71b2\") " pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:39.936293 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:39.936267    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txv2k\""
Jul 20 09:59:39.944117 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:39.944097    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:40.172101 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:40.172057    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94j7v"]
Jul 20 09:59:40.174955 ip-10-0-143-155 kubenswrapper[2546]: W0720 09:59:40.174926    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb3e353_a052_4603_917b_754e8fef71b2.slice/crio-2f6d20832ee41c427308389433ab516d44d4f180d5c72d50e3e71d3e500aa659 WatchSource:0}: Error finding container 2f6d20832ee41c427308389433ab516d44d4f180d5c72d50e3e71d3e500aa659: Status 404 returned error can't find the container with id 2f6d20832ee41c427308389433ab516d44d4f180d5c72d50e3e71d3e500aa659
Jul 20 09:59:40.755916 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:40.755870    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j7v" event={"ID":"2eb3e353-a052-4603-917b-754e8fef71b2","Type":"ContainerStarted","Data":"2f6d20832ee41c427308389433ab516d44d4f180d5c72d50e3e71d3e500aa659"}
Jul 20 09:59:42.565956 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.565923    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:42.566392 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.565978    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 09:59:42.765402 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.765317    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j7v" event={"ID":"2eb3e353-a052-4603-917b-754e8fef71b2","Type":"ContainerStarted","Data":"fe6f605ae6cc3cc081d883ec3644d1295c799f9c68ce7b7aaa2057fa4ede8ad9"}
Jul 20 09:59:42.765402 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.765372    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j7v" event={"ID":"2eb3e353-a052-4603-917b-754e8fef71b2","Type":"ContainerStarted","Data":"9d2e3ea6932bec28dde5579760d58822b963c6dd06901959537e3bbff05e313f"}
Jul 20 09:59:42.765699 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.765488    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:42.785028 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:42.784986    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-94j7v" podStartSLOduration=129.816204743 podStartE2EDuration="2m11.78497438s" totalImagesPullingTime="1.968769637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 09:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 09:59:42.783628562 +0000 UTC m=+166.831053318" watchObservedRunningTime="2026-07-20 09:59:42.78497438 +0000 UTC m=+166.832399125"
Jul 20 09:59:44.072342 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:44.072305    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:44.072805 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:44.072374    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:44.077394 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:44.077376    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:44.778323 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:44.778294    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 09:59:44.933611 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:44.933571    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:45.001034 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:45.000995    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:45.065276 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:45.065185    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:45.496699 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:45.496668    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:45.564130 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:45.564093    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:51.806152 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:51.806110    2546 generic.go:356] "Generic (PLEG): container finished" podID="2b0979e2-1e37-4d9a-a017-3771bf4a5d27" containerID="7cd5e0d743022e76da2788f778df1913cfd554b2e16604e33d7e262251e6773d" exitCode=0
Jul 20 09:59:51.806650 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:51.806154    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-c54fp" event={"ID":"2b0979e2-1e37-4d9a-a017-3771bf4a5d27","Type":"ContainerDied","Data":"7cd5e0d743022e76da2788f778df1913cfd554b2e16604e33d7e262251e6773d"}
Jul 20 09:59:51.806650 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:51.806557    2546 scope.go:117] "RemoveContainer" containerID="7cd5e0d743022e76da2788f778df1913cfd554b2e16604e33d7e262251e6773d"
Jul 20 09:59:52.771713 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:52.771681    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94j7v"
Jul 20 09:59:52.809587 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:52.809558    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j7v_2eb3e353-a052-4603-917b-754e8fef71b2/dns/0.log"
Jul 20 09:59:52.814532 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:52.814502    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-c54fp" event={"ID":"2b0979e2-1e37-4d9a-a017-3771bf4a5d27","Type":"ContainerStarted","Data":"d4667fb823716b03964a7980c9bd59af66d9d5abb23342202f2be00bbf5560f3"}
Jul 20 09:59:52.817383 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:52.817362    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j7v_2eb3e353-a052-4603-917b-754e8fef71b2/kube-rbac-proxy/0.log"
Jul 20 09:59:52.914157 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:52.914129    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4l429_df265ea5-1894-4478-8759-17110f267128/dns-node-resolver/0.log"
Jul 20 09:59:53.709607 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:53.709563    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76f66965bf-qmdng" podUID="e8738652-cc9a-4df3-981c-4c329a13baff" containerName="console" containerID="cri-o://75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817" gracePeriod=15
Jul 20 09:59:53.979206 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:53.979150    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:54.038133 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038110    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038254 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038146    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038301 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038282    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038358 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038346    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038398 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038370    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038398 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038394    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k88mm\" (UniqueName: \"kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038531 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038450    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config\") pod \"e8738652-cc9a-4df3-981c-4c329a13baff\" (UID: \"e8738652-cc9a-4df3-981c-4c329a13baff\") "
Jul 20 09:59:54.038531 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038508    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:54.038737 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038718    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-oauth-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.038821 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038740    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca" (OuterVolumeSpecName: "service-ca") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:54.038821 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038753    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:54.038821 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.038801    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config" (OuterVolumeSpecName: "console-config") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 09:59:54.040407 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.040375    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:54.040524 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.040476    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 09:59:54.040524 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.040474    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm" (OuterVolumeSpecName: "kube-api-access-k88mm") pod "e8738652-cc9a-4df3-981c-4c329a13baff" (UID: "e8738652-cc9a-4df3-981c-4c329a13baff"). InnerVolumeSpecName "kube-api-access-k88mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 09:59:54.139394 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139370    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-service-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.139394 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139390    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.139518 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139400    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88mm\" (UniqueName: \"kubernetes.io/projected/e8738652-cc9a-4df3-981c-4c329a13baff-kube-api-access-k88mm\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.139518 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139409    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-console-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.139518 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139418    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8738652-cc9a-4df3-981c-4c329a13baff-trusted-ca-bundle\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.139518 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.139427    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8738652-cc9a-4df3-981c-4c329a13baff-console-oauth-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 09:59:54.823968 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.823891    2546 generic.go:356] "Generic (PLEG): container finished" podID="e8738652-cc9a-4df3-981c-4c329a13baff" containerID="75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817" exitCode=0
Jul 20 09:59:54.824131 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.823966    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f66965bf-qmdng"
Jul 20 09:59:54.824131 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.823985    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f66965bf-qmdng" event={"ID":"e8738652-cc9a-4df3-981c-4c329a13baff","Type":"ContainerDied","Data":"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"}
Jul 20 09:59:54.824131 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.824030    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f66965bf-qmdng" event={"ID":"e8738652-cc9a-4df3-981c-4c329a13baff","Type":"ContainerDied","Data":"896c900adcb7d71d17a90519d0d1b6cd89fe36ab9747a0d1f39bf92568faaa11"}
Jul 20 09:59:54.824131 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.824052    2546 scope.go:117] "RemoveContainer" containerID="75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"
Jul 20 09:59:54.839880 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.839840    2546 scope.go:117] "RemoveContainer" containerID="75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"
Jul 20 09:59:54.840180 ip-10-0-143-155 kubenswrapper[2546]: E0720 09:59:54.840139    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817\": container with ID starting with 75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817 not found: ID does not exist" containerID="75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"
Jul 20 09:59:54.840318 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.840179    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817"} err="failed to get container status \"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817\": rpc error: code = NotFound desc = could not find container \"75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817\": container with ID starting with 75000b83a170a954445ad21f3850d30a0b73dad9be8acb5b431c643516f95817 not found: ID does not exist"
Jul 20 09:59:54.841998 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.841979    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:54.849268 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:54.849248    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76f66965bf-qmdng"]
Jul 20 09:59:56.402956 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:56.402923    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8738652-cc9a-4df3-981c-4c329a13baff" path="/var/lib/kubelet/pods/e8738652-cc9a-4df3-981c-4c329a13baff/volumes"
Jul 20 09:59:56.839408 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:56.839377    2546 generic.go:356] "Generic (PLEG): container finished" podID="a02fa1cc-3cf8-46d8-bf8d-351f97251ded" containerID="90b3365aec2c46f8cad56268ae3686771bdafdeddaada4998d9f5532327fdf0e" exitCode=0
Jul 20 09:59:56.839642 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:56.839417    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj" event={"ID":"a02fa1cc-3cf8-46d8-bf8d-351f97251ded","Type":"ContainerDied","Data":"90b3365aec2c46f8cad56268ae3686771bdafdeddaada4998d9f5532327fdf0e"}
Jul 20 09:59:56.839741 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:56.839708    2546 scope.go:117] "RemoveContainer" containerID="90b3365aec2c46f8cad56268ae3686771bdafdeddaada4998d9f5532327fdf0e"
Jul 20 09:59:57.144617 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:57.144537    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:57.202092 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:57.202051    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 09:59:57.845893 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:57.845861    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-tzcfj" event={"ID":"a02fa1cc-3cf8-46d8-bf8d-351f97251ded","Type":"ContainerStarted","Data":"515f5745e8933d85f54c4e6481d5a91646135b7f45526007610947ee972ce052"}
Jul 20 09:59:59.747705 ip-10-0-143-155 kubenswrapper[2546]: I0720 09:59:59.747640    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fbc9ff54-zs7x4" podUID="8adc1968-b4cc-465a-b0f7-12871dadbee6" containerName="console" containerID="cri-o://4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a" gracePeriod=15
Jul 20 10:00:00.021310 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.021244    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 10:00:00.085743 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085720    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxxcd\" (UniqueName: \"kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.085877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085762    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.085877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085808    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.085877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085829    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.085877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085865    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.086113 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.085953    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.086113 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.086012    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config\") pod \"8adc1968-b4cc-465a-b0f7-12871dadbee6\" (UID: \"8adc1968-b4cc-465a-b0f7-12871dadbee6\") "
Jul 20 10:00:00.086299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.086268    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:00.086373 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.086291    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:00.086373 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.086305    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca" (OuterVolumeSpecName: "service-ca") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:00.086473 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.086398    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config" (OuterVolumeSpecName: "console-config") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:00.087893 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.087861    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd" (OuterVolumeSpecName: "kube-api-access-rxxcd") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "kube-api-access-rxxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:00:00.088002 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.087909    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:00.088002 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.087951    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8adc1968-b4cc-465a-b0f7-12871dadbee6" (UID: "8adc1968-b4cc-465a-b0f7-12871dadbee6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:00.187061 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187034    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-service-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187061 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187056    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-oauth-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187088    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-trusted-ca-bundle\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187099    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-oauth-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187107    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187115    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxcd\" (UniqueName: \"kubernetes.io/projected/8adc1968-b4cc-465a-b0f7-12871dadbee6-kube-api-access-rxxcd\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.187273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.187124    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8adc1968-b4cc-465a-b0f7-12871dadbee6-console-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:00.860521 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.860489    2546 generic.go:356] "Generic (PLEG): container finished" podID="8adc1968-b4cc-465a-b0f7-12871dadbee6" containerID="4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a" exitCode=0
Jul 20 10:00:00.860892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.860534    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fbc9ff54-zs7x4"
Jul 20 10:00:00.860892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.860576    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc9ff54-zs7x4" event={"ID":"8adc1968-b4cc-465a-b0f7-12871dadbee6","Type":"ContainerDied","Data":"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"}
Jul 20 10:00:00.860892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.860629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fbc9ff54-zs7x4" event={"ID":"8adc1968-b4cc-465a-b0f7-12871dadbee6","Type":"ContainerDied","Data":"4429e5cff57bb68a55f43182dfc92529126758515a2ec1f13f314ca1adc52734"}
Jul 20 10:00:00.860892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.860655    2546 scope.go:117] "RemoveContainer" containerID="4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"
Jul 20 10:00:00.870970 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.870936    2546 scope.go:117] "RemoveContainer" containerID="4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"
Jul 20 10:00:00.871542 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:00.871514    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a\": container with ID starting with 4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a not found: ID does not exist" containerID="4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"
Jul 20 10:00:00.871616 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.871549    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a"} err="failed to get container status \"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a\": rpc error: code = NotFound desc = could not find container \"4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a\": container with ID starting with 4786209bcec7cbf03344393416e417701547c6b32de6b63c16da5a2e8fc9a02a not found: ID does not exist"
Jul 20 10:00:00.877795 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.877774    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 10:00:00.883664 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:00.883643    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fbc9ff54-zs7x4"]
Jul 20 10:00:02.402626 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:02.402593    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adc1968-b4cc-465a-b0f7-12871dadbee6" path="/var/lib/kubelet/pods/8adc1968-b4cc-465a-b0f7-12871dadbee6/volumes"
Jul 20 10:00:02.570533 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:02.570506    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 10:00:02.574274 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:02.574252    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75b788647d-bb5bd"
Jul 20 10:00:06.309545 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:06.309504    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:06.508546 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:06.508476    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:23.795188 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:23.795129    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:34.030582 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:34.030542    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:34.096157 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:34.096125    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:39.027835 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:39.027806    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:00:39.028224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:39.027845    2546 generic.go:356] "Generic (PLEG): container finished" podID="7be60d37-9c23-4eb3-bfcc-c5688f5ee538" containerID="d9e241c9ef59360301c8dc10a1a5be0473b04371586e1389a013fae26f2cbb43" exitCode=2
Jul 20 10:00:39.028224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:39.027921    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr" event={"ID":"7be60d37-9c23-4eb3-bfcc-c5688f5ee538","Type":"ContainerDied","Data":"d9e241c9ef59360301c8dc10a1a5be0473b04371586e1389a013fae26f2cbb43"}
Jul 20 10:00:39.028299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:39.028230    2546 scope.go:117] "RemoveContainer" containerID="d9e241c9ef59360301c8dc10a1a5be0473b04371586e1389a013fae26f2cbb43"
Jul 20 10:00:40.034046 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:40.034015    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:00:40.034542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:40.034132    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-m5ggr" event={"ID":"7be60d37-9c23-4eb3-bfcc-c5688f5ee538","Type":"ContainerStarted","Data":"bac02881ea6cd230492f40a44fd5ab3c2457de12d87a813ae6dcb715fa3c6fdc"}
Jul 20 10:00:41.659258 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:41.659193    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:41.716833 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:41.716801    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:43.519226 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519191    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:43.519617 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519455    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="alertmanager" containerID="cri-o://941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291" gracePeriod=120
Jul 20 10:00:43.519617 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519525    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-metric" containerID="cri-o://a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477" gracePeriod=120
Jul 20 10:00:43.519617 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519574    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="prom-label-proxy" containerID="cri-o://5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358" gracePeriod=120
Jul 20 10:00:43.519776 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519598    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-web" containerID="cri-o://6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a" gracePeriod=120
Jul 20 10:00:43.519776 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519619    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy" containerID="cri-o://2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a" gracePeriod=120
Jul 20 10:00:43.519776 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:43.519634    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="config-reloader" containerID="cri-o://8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b" gracePeriod=120
Jul 20 10:00:44.053737 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053706    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358" exitCode=0
Jul 20 10:00:44.053737 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053729    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477" exitCode=0
Jul 20 10:00:44.053737 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053735    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a" exitCode=0
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053742    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b" exitCode=0
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053758    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291" exitCode=0
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053781    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"}
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053854    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"}
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053882    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"}
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053905    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"}
Jul 20 10:00:44.053969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.053929    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"}
Jul 20 10:00:44.765297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.765276    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:44.937988 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.937957    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.937988 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.937994    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938010    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938051    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kb8j\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938101    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938127    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938150    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938187    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938234 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938228    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938250    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938277    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938314    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938341    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config\") pod \"29a16b8a-5447-4c09-b4a8-f879ebf40504\" (UID: \"29a16b8a-5447-4c09-b4a8-f879ebf40504\") "
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938364    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:00:44.938589 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938423    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:44.938903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938658    2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:44.938903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938785    2546 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-alertmanager-main-db\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:44.938903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.938884    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:00:44.941025 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941001    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out" (OuterVolumeSpecName: "config-out") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:00:44.941329 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941301    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.941414 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941358    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j" (OuterVolumeSpecName: "kube-api-access-9kb8j") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "kube-api-access-9kb8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:00:44.941508 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941403    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.941567 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941548    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume" (OuterVolumeSpecName: "config-volume") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.941816 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.941789    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.943030 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.943007    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:00:44.943495 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.943472    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.945513 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.945491    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:44.952185 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:44.952162    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config" (OuterVolumeSpecName: "web-config") pod "29a16b8a-5447-4c09-b4a8-f879ebf40504" (UID: "29a16b8a-5447-4c09-b4a8-f879ebf40504"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:00:45.040115 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040063    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040115 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040114    2546 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-tls-assets\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040115 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040125    2546 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-out\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040134    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040142    2546 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-config-volume\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040171    2546 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-web-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040179    2546 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-cluster-tls-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040187    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kb8j\" (UniqueName: \"kubernetes.io/projected/29a16b8a-5447-4c09-b4a8-f879ebf40504-kube-api-access-9kb8j\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040196    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-main-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040204    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29a16b8a-5447-4c09-b4a8-f879ebf40504-metrics-client-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.040290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.040212    2546 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/29a16b8a-5447-4c09-b4a8-f879ebf40504-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:00:45.060527 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.060500    2546 generic.go:356] "Generic (PLEG): container finished" podID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerID="6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a" exitCode=0
Jul 20 10:00:45.060618 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.060578    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"}
Jul 20 10:00:45.060618 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.060594    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.060685 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.060665    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"29a16b8a-5447-4c09-b4a8-f879ebf40504","Type":"ContainerDied","Data":"ff18df72df8236208e127927b325c2a3c1db70b96aaaff77eb9b9aeda6868787"}
Jul 20 10:00:45.060718 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.060698    2546 scope.go:117] "RemoveContainer" containerID="5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"
Jul 20 10:00:45.068457 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.068438    2546 scope.go:117] "RemoveContainer" containerID="a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"
Jul 20 10:00:45.075435 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.075418    2546 scope.go:117] "RemoveContainer" containerID="2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"
Jul 20 10:00:45.081788 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.081772    2546 scope.go:117] "RemoveContainer" containerID="6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"
Jul 20 10:00:45.084800 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.084781    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:45.088974 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.088955    2546 scope.go:117] "RemoveContainer" containerID="8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"
Jul 20 10:00:45.089901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.089884    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:45.095711 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.095694    2546 scope.go:117] "RemoveContainer" containerID="941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"
Jul 20 10:00:45.102381 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.102367    2546 scope.go:117] "RemoveContainer" containerID="754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a"
Jul 20 10:00:45.108890 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.108874    2546 scope.go:117] "RemoveContainer" containerID="5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"
Jul 20 10:00:45.109139 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.109121    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358\": container with ID starting with 5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358 not found: ID does not exist" containerID="5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"
Jul 20 10:00:45.109206 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109146    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358"} err="failed to get container status \"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358\": rpc error: code = NotFound desc = could not find container \"5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358\": container with ID starting with 5687e54883b0c1c144c6247290b63e4a41ebe0b7cd77f54fd3a37c1189f54358 not found: ID does not exist"
Jul 20 10:00:45.109206 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109163    2546 scope.go:117] "RemoveContainer" containerID="a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"
Jul 20 10:00:45.109383 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.109367    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477\": container with ID starting with a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477 not found: ID does not exist" containerID="a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"
Jul 20 10:00:45.109419 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109387    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477"} err="failed to get container status \"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477\": rpc error: code = NotFound desc = could not find container \"a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477\": container with ID starting with a483e34a4f357c5c707756906f137a7811e614768b9a03d30631ff8ca2505477 not found: ID does not exist"
Jul 20 10:00:45.109419 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109403    2546 scope.go:117] "RemoveContainer" containerID="2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"
Jul 20 10:00:45.109604 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.109589    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a\": container with ID starting with 2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a not found: ID does not exist" containerID="2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"
Jul 20 10:00:45.109643 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109609    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a"} err="failed to get container status \"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a\": rpc error: code = NotFound desc = could not find container \"2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a\": container with ID starting with 2365d320e9949a4ca34b8445a6e6f79e3efbf9cc5e9cd4250834a1c660d01f9a not found: ID does not exist"
Jul 20 10:00:45.109643 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109629    2546 scope.go:117] "RemoveContainer" containerID="6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"
Jul 20 10:00:45.109805 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.109790    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a\": container with ID starting with 6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a not found: ID does not exist" containerID="6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"
Jul 20 10:00:45.109844 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109808    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a"} err="failed to get container status \"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a\": rpc error: code = NotFound desc = could not find container \"6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a\": container with ID starting with 6194229788ed4defab921450cb7575abdf6732c5737e613ee79c291207b33e2a not found: ID does not exist"
Jul 20 10:00:45.109844 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109820    2546 scope.go:117] "RemoveContainer" containerID="8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"
Jul 20 10:00:45.109963 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.109949    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b\": container with ID starting with 8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b not found: ID does not exist" containerID="8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"
Jul 20 10:00:45.110006 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109965    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b"} err="failed to get container status \"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b\": rpc error: code = NotFound desc = could not find container \"8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b\": container with ID starting with 8e15bfa9ede0cacdee6802a435d345f217379aa9150899eb7958d5543792ae2b not found: ID does not exist"
Jul 20 10:00:45.110006 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.109981    2546 scope.go:117] "RemoveContainer" containerID="941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"
Jul 20 10:00:45.110237 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.110217    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291\": container with ID starting with 941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291 not found: ID does not exist" containerID="941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"
Jul 20 10:00:45.110288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.110247    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291"} err="failed to get container status \"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291\": rpc error: code = NotFound desc = could not find container \"941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291\": container with ID starting with 941986af109e35b2a70caebc04e8f8d367948c150d4846c8607e0b9d8a7dc291 not found: ID does not exist"
Jul 20 10:00:45.110288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.110262    2546 scope.go:117] "RemoveContainer" containerID="754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a"
Jul 20 10:00:45.110492 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:00:45.110475    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a\": container with ID starting with 754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a not found: ID does not exist" containerID="754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a"
Jul 20 10:00:45.110544 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.110500    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a"} err="failed to get container status \"754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a\": rpc error: code = NotFound desc = could not find container \"754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a\": container with ID starting with 754aa2453179ee6186200138bd5b1ddcb51932d15f9ba17c163878a43f1a9b7a not found: ID does not exist"
Jul 20 10:00:45.121842 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.121823    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:45.122177 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122162    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="init-config-reloader"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122180    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="init-config-reloader"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122191    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="config-reloader"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122199    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="config-reloader"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122213    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adc1968-b4cc-465a-b0f7-12871dadbee6" containerName="console"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122221    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adc1968-b4cc-465a-b0f7-12871dadbee6" containerName="console"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122236    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="alertmanager"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122244    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="alertmanager"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122259    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-web"
Jul 20 10:00:45.122267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122267    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-web"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122283    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8738652-cc9a-4df3-981c-4c329a13baff" containerName="console"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122293    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8738652-cc9a-4df3-981c-4c329a13baff" containerName="console"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122304    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122312    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122331    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-metric"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122339    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-metric"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122349    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="prom-label-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122358    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="prom-label-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122435    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8738652-cc9a-4df3-981c-4c329a13baff" containerName="console"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122447    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adc1968-b4cc-465a-b0f7-12871dadbee6" containerName="console"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122456    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122466    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="alertmanager"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122475    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-metric"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122484    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="prom-label-proxy"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122493    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="kube-rbac-proxy-web"
Jul 20 10:00:45.122696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.122502    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" containerName="config-reloader"
Jul 20 10:00:45.126630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.126612    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.129708 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129689    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 20 10:00:45.129708 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129701    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qp8dt\""
Jul 20 10:00:45.129842 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129813    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 20 10:00:45.129842 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129828    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 20 10:00:45.129924 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129846    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 20 10:00:45.129924 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.129816    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 20 10:00:45.130130 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.130114    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 20 10:00:45.130207 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.130117    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 20 10:00:45.130350 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.130334    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 20 10:00:45.135443 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.135424    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 20 10:00:45.143294 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.143275    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:45.241653 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241589    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241653 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241621    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241789 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241655    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241789 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241694    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241789 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241732    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2dm\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-kube-api-access-cg2dm\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241789 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241753    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241789 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241774    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241792    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-web-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241845    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-out\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241875    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241895    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.241955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.242158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.241979    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342596 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342570    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342609    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2dm\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-kube-api-access-cg2dm\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342628    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342648    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342663    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-web-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342686    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-out\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342706    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342722    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342741    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342784    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342825    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.342914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.342898    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.344169 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.343441    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.344169 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.343607    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.344169 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.343912    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.345856 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.345822    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.345856 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.345842    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346019 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.345967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346019 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.345971    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-web-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346192 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.346173    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346237 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.346218    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-out\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346341 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.346319    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.346847 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.346831    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.347195 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.347159    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7dfd8a09-9c00-4af8-bde3-205b318447f5-config-volume\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.351417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.351400    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2dm\" (UniqueName: \"kubernetes.io/projected/7dfd8a09-9c00-4af8-bde3-205b318447f5-kube-api-access-cg2dm\") pod \"alertmanager-main-0\" (UID: \"7dfd8a09-9c00-4af8-bde3-205b318447f5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.435448 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.435428    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 20 10:00:45.573326 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.573302    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 20 10:00:45.577383 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:00:45.577355    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfd8a09_9c00_4af8_bde3_205b318447f5.slice/crio-7938dfdfd4142b63c0536b74d686e442ca46696522c1f8472a335dfb4608a4ef WatchSource:0}: Error finding container 7938dfdfd4142b63c0536b74d686e442ca46696522c1f8472a335dfb4608a4ef: Status 404 returned error can't find the container with id 7938dfdfd4142b63c0536b74d686e442ca46696522c1f8472a335dfb4608a4ef
Jul 20 10:00:45.941346 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:45.941316    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:46.008624 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:46.008595    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:46.067813 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:46.067780    2546 generic.go:356] "Generic (PLEG): container finished" podID="7dfd8a09-9c00-4af8-bde3-205b318447f5" containerID="766e524e63ed43c5fccd114d994eae4ea2578118084bb0a1f58ae606353555c4" exitCode=0
Jul 20 10:00:46.067927 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:46.067868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerDied","Data":"766e524e63ed43c5fccd114d994eae4ea2578118084bb0a1f58ae606353555c4"}
Jul 20 10:00:46.067971 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:46.067943    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"7938dfdfd4142b63c0536b74d686e442ca46696522c1f8472a335dfb4608a4ef"}
Jul 20 10:00:46.403427 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:46.403393    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a16b8a-5447-4c09-b4a8-f879ebf40504" path="/var/lib/kubelet/pods/29a16b8a-5447-4c09-b4a8-f879ebf40504/volumes"
Jul 20 10:00:47.076238 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"ee2b63cfc55b64573714258dfa52d2f23b766b7b36f469e709e33b9616172d53"}
Jul 20 10:00:47.076670 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076267    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"85698846dbd75b16b5167ecbdc816c97f0ddc0db0512b09a15cc90f2527c9f2e"}
Jul 20 10:00:47.076670 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076295    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"a3ede31d307c61d75cd4d2faf012c6952ad671cb24042d3b902b6ab1e73977a3"}
Jul 20 10:00:47.076670 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076323    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"dbe72e38faa1db84494cb2544f898058f511dfbd4a0a2d3ed0d7ca9eb60d83b7"}
Jul 20 10:00:47.076670 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076347    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"a6c8136bd7bbf8ad85f6cf30c1140d5dfc9b012da33ad66a8184bc58f7dc8d5e"}
Jul 20 10:00:47.076670 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:47.076369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7dfd8a09-9c00-4af8-bde3-205b318447f5","Type":"ContainerStarted","Data":"71f99c63948c9aba855f95ac7504ef3424ef3878f4e210e4061b5ad7cea7fc9b"}
Jul 20 10:00:52.648635 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:52.648575    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.633012227 podStartE2EDuration="7.648556528s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.544301ms" isStatefulPod=false podCreationTimestamp="2026-07-20 10:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:00:47.122553352 +0000 UTC m=+231.169978096" watchObservedRunningTime="2026-07-20 10:00:52.648556528 +0000 UTC m=+236.695981274"
Jul 20 10:00:52.649600 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:52.649576    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 10:00:52.757284 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:52.757256    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:52.857377 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:52.857343    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:00:52.957967 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:00:52.957891    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:00.490981 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:00.490944    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:00.589766 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:00.589733    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:17.669974 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:17.669908    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67c4f65cc-r9k2q" podUID="0c7b9606-e139-404e-881f-2340d8005916" containerName="console" containerID="cri-o://751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4" gracePeriod=15
Jul 20 10:01:17.914680 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:17.914658    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 10:01:18.003489 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003414    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003489 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003449    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003489 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003490    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zwfz\" (UniqueName: \"kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003506    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003537    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003580    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.003736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003596    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config\") pod \"0c7b9606-e139-404e-881f-2340d8005916\" (UID: \"0c7b9606-e139-404e-881f-2340d8005916\") "
Jul 20 10:01:18.004053 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.003997    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca" (OuterVolumeSpecName: "service-ca") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:18.004053 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.004010    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:18.004053 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.004027    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:18.004240 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.004102    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config" (OuterVolumeSpecName: "console-config") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:01:18.005619 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.005597    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz" (OuterVolumeSpecName: "kube-api-access-6zwfz") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "kube-api-access-6zwfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:01:18.006001 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.005984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:01:18.006087 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.006000    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0c7b9606-e139-404e-881f-2340d8005916" (UID: "0c7b9606-e139-404e-881f-2340d8005916"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:01:18.104158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104135    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zwfz\" (UniqueName: \"kubernetes.io/projected/0c7b9606-e139-404e-881f-2340d8005916-kube-api-access-6zwfz\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104156    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-oauth-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104165    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-service-ca\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104174    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-oauth-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104182    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-console-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104190    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c7b9606-e139-404e-881f-2340d8005916-console-serving-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.104290 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.104199    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c7b9606-e139-404e-881f-2340d8005916-trusted-ca-bundle\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:01:18.204266 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.204232    2546 generic.go:356] "Generic (PLEG): container finished" podID="0c7b9606-e139-404e-881f-2340d8005916" containerID="751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4" exitCode=0
Jul 20 10:01:18.204417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.204283    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67c4f65cc-r9k2q"
Jul 20 10:01:18.204417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.204275    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c4f65cc-r9k2q" event={"ID":"0c7b9606-e139-404e-881f-2340d8005916","Type":"ContainerDied","Data":"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"}
Jul 20 10:01:18.204417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.204402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67c4f65cc-r9k2q" event={"ID":"0c7b9606-e139-404e-881f-2340d8005916","Type":"ContainerDied","Data":"98d98fb70ed9a99fd65c4c7e09901a7faa8328cdfd1491db665f139b6ec489c9"}
Jul 20 10:01:18.204581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.204430    2546 scope.go:117] "RemoveContainer" containerID="751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"
Jul 20 10:01:18.213162 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.213145    2546 scope.go:117] "RemoveContainer" containerID="751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"
Jul 20 10:01:18.213431 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:01:18.213413    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4\": container with ID starting with 751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4 not found: ID does not exist" containerID="751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"
Jul 20 10:01:18.213506 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.213437    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4"} err="failed to get container status \"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4\": rpc error: code = NotFound desc = could not find container \"751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4\": container with ID starting with 751f7da9e65548e91495fa8a0f89dfba1e430cff5b8f48956596483271f6cdb4 not found: ID does not exist"
Jul 20 10:01:18.226829 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.226803    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 10:01:18.230801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.230778    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67c4f65cc-r9k2q"]
Jul 20 10:01:18.402536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:18.402499    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c7b9606-e139-404e-881f-2340d8005916" path="/var/lib/kubelet/pods/0c7b9606-e139-404e-881f-2340d8005916/volumes"
Jul 20 10:01:22.983094 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:22.982996    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:23.067989 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:23.067956    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:23.972078 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:23.972025    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:24.070567 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:24.070532    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:01:56.338377 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:56.338345    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:01:56.339526 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:56.339506    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:01:56.346158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:56.346136    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:01:56.347610 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:56.347591    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:01:56.354407 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:56.354389    2546 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 10:01:57.297313 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:01:57.297274    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:37.052021 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:02:37.051987    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:02:37.119534 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:02:37.119499    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:14.668224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:03:14.668146    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:03:16.430832 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:03:16.430798    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:03.420536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:03.420480    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:03.486955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:03.486923    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:03.813710 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:03.813622    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:03.916824 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:03.916784    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:03.977093 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:03.977052    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:26.355523 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:26.355485    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:26.778430 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:26.778333    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:27.023376 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:27.023343    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:29.156491 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.156456    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"]
Jul 20 10:04:29.156897 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.156769    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c7b9606-e139-404e-881f-2340d8005916" containerName="console"
Jul 20 10:04:29.156897 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.156779    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c7b9606-e139-404e-881f-2340d8005916" containerName="console"
Jul 20 10:04:29.156897 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.156853    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c7b9606-e139-404e-881f-2340d8005916" containerName="console"
Jul 20 10:04:29.159746 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.159728    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.163176 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.163154    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 20 10:04:29.163312 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.163294    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 20 10:04:29.163396 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.163375    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4wq8g\""
Jul 20 10:04:29.163962 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.163946    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 20 10:04:29.177251 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.177226    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"]
Jul 20 10:04:29.296407 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.296375    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lhz\" (UniqueName: \"kubernetes.io/projected/5658f5dd-e3ce-4419-9efc-6860d71cc311-kube-api-access-64lhz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.296554 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.296436    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5658f5dd-e3ce-4419-9efc-6860d71cc311-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.397879 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.397844    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5658f5dd-e3ce-4419-9efc-6860d71cc311-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.398044 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.397934    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lhz\" (UniqueName: \"kubernetes.io/projected/5658f5dd-e3ce-4419-9efc-6860d71cc311-kube-api-access-64lhz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.400296 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.400266    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5658f5dd-e3ce-4419-9efc-6860d71cc311-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.436177 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.436119    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lhz\" (UniqueName: \"kubernetes.io/projected/5658f5dd-e3ce-4419-9efc-6860d71cc311-kube-api-access-64lhz\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-84jl5\" (UID: \"5658f5dd-e3ce-4419-9efc-6860d71cc311\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.469910 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.469877    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:29.627575 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.627545    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"]
Jul 20 10:04:29.636841 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:04:29.636817    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5658f5dd_e3ce_4419_9efc_6860d71cc311.slice/crio-dc091130a05c9af9fbd39ab55d0e05ca9d9770ca689ad68f416c22b69290303c WatchSource:0}: Error finding container dc091130a05c9af9fbd39ab55d0e05ca9d9770ca689ad68f416c22b69290303c: Status 404 returned error can't find the container with id dc091130a05c9af9fbd39ab55d0e05ca9d9770ca689ad68f416c22b69290303c
Jul 20 10:04:29.639235 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.639218    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:04:29.967061 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:29.967032    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5" event={"ID":"5658f5dd-e3ce-4419-9efc-6860d71cc311","Type":"ContainerStarted","Data":"dc091130a05c9af9fbd39ab55d0e05ca9d9770ca689ad68f416c22b69290303c"}
Jul 20 10:04:32.989174 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:32.989089    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5" event={"ID":"5658f5dd-e3ce-4419-9efc-6860d71cc311","Type":"ContainerStarted","Data":"a98d5754b058787cbdaaa0ba3908d8daf9f3ee6eab50c1dda2b944aa32cd9552"}
Jul 20 10:04:32.989555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:32.989214    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:33.081488 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:33.081446    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5" podStartSLOduration=1.092896301 podStartE2EDuration="4.081428839s" totalImagesPullingTime="2.988532538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:04:33.066863794 +0000 UTC m=+457.114288517" watchObservedRunningTime="2026-07-20 10:04:33.081428839 +0000 UTC m=+457.128853585"
Jul 20 10:04:33.356342 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:33.356306    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:33.947366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:33.947303    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:34.107838 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:34.107804    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:34.933426 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:34.933389    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.038002 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.037949    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.152168 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.152135    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.261993 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.261919    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.537147 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.537053    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.647419 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.647384    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:35.851145 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:35.851117    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:36.078008 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:36.077966    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:47.382120 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.382084    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-795b5d4c5f-58cds"]
Jul 20 10:04:47.390943 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.390908    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.395807 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.395774    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 10:04:47.395975 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.395840    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 10:04:47.396103 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.396086    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 10:04:47.396462 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.396434    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 10:04:47.396572 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.396517    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7mwdt\""
Jul 20 10:04:47.397145 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.397131    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 10:04:47.409201 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.409179    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795b5d4c5f-58cds"]
Jul 20 10:04:47.418049 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.418027    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 10:04:47.507754 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.507729    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:47.539611 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539590    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-service-ca\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539746 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-trusted-ca-bundle\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539746 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539678    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-oauth-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539746 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539734    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-oauth-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539758    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539806    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-console-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.539903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.539830    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5nd\" (UniqueName: \"kubernetes.io/projected/59bdb034-d7d9-4494-9a46-7967e6e17099-kube-api-access-kf5nd\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.623010 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.622985    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:47.641196 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-service-ca\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641196    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-trusted-ca-bundle\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641219    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-oauth-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641253    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-oauth-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641276    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641512 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641300    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-console-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641512 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641325    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5nd\" (UniqueName: \"kubernetes.io/projected/59bdb034-d7d9-4494-9a46-7967e6e17099-kube-api-access-kf5nd\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.641906 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.641878    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-service-ca\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.642052 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.642031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-oauth-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.642157 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.642053    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-trusted-ca-bundle\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.642157 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.642135    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59bdb034-d7d9-4494-9a46-7967e6e17099-console-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.643910 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.643892    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-serving-cert\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.643989 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.643891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59bdb034-d7d9-4494-9a46-7967e6e17099-console-oauth-config\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.652669 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.652648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5nd\" (UniqueName: \"kubernetes.io/projected/59bdb034-d7d9-4494-9a46-7967e6e17099-kube-api-access-kf5nd\") pod \"console-795b5d4c5f-58cds\" (UID: \"59bdb034-d7d9-4494-9a46-7967e6e17099\") " pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.700276 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.700250    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:47.843243 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:47.843207    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795b5d4c5f-58cds"]
Jul 20 10:04:47.847719 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:04:47.847693    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bdb034_d7d9_4494_9a46_7967e6e17099.slice/crio-e8f1fa7b375aba4f9e2beb2733eba4a37854ed2505ae2bc91164b5dc3295ba68 WatchSource:0}: Error finding container e8f1fa7b375aba4f9e2beb2733eba4a37854ed2505ae2bc91164b5dc3295ba68: Status 404 returned error can't find the container with id e8f1fa7b375aba4f9e2beb2733eba4a37854ed2505ae2bc91164b5dc3295ba68
Jul 20 10:04:48.053892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:48.053859    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795b5d4c5f-58cds" event={"ID":"59bdb034-d7d9-4494-9a46-7967e6e17099","Type":"ContainerStarted","Data":"e37403d5102edd53b9687706df53718b8cc0f94cc0ef0f3cb6705c1aa207f876"}
Jul 20 10:04:48.054046 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:48.053904    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795b5d4c5f-58cds" event={"ID":"59bdb034-d7d9-4494-9a46-7967e6e17099","Type":"ContainerStarted","Data":"e8f1fa7b375aba4f9e2beb2733eba4a37854ed2505ae2bc91164b5dc3295ba68"}
Jul 20 10:04:48.077161 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:48.077110    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-795b5d4c5f-58cds" podStartSLOduration=1.077094695 podStartE2EDuration="1.077094695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:04:48.075135045 +0000 UTC m=+472.122559789" watchObservedRunningTime="2026-07-20 10:04:48.077094695 +0000 UTC m=+472.124519440"
Jul 20 10:04:53.995823 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:53.995793    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-84jl5"
Jul 20 10:04:57.700682 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:57.700648    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:57.701114 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:57.700698    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:57.705424 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:57.705401    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:58.102978 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:58.102948    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-795b5d4c5f-58cds"
Jul 20 10:04:58.325437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:58.325400    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:04:58.435267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:04:58.435187    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:55.933502 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:55.933416    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:56.021114 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.021087    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:05:56.023392 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.023376    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.026531 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.026497    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 20 10:05:56.026658 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.026503    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 20 10:05:56.027653 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.027633    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 20 10:05:56.027766 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.027633    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-lh5w6\""
Jul 20 10:05:56.046877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.046847    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:05:56.069276 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.069252    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:56.109718 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.109691    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nml64"]
Jul 20 10:05:56.111935 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.111918    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.116010 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.115989    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-89b4t\""
Jul 20 10:05:56.116122 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.115989    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:05:56.127707 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.127686    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nml64"]
Jul 20 10:05:56.166350 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.166324    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.166509 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.166401    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfl5\" (UniqueName: \"kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.200819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.200751    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:56.267413 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.267388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xg8\" (UniqueName: \"kubernetes.io/projected/d4ae3989-afef-4444-86a0-c4f27df04db5-kube-api-access-98xg8\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.267517 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.267433    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.267517 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.267494    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4ae3989-afef-4444-86a0-c4f27df04db5-data\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.267629 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.267533    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfl5\" (UniqueName: \"kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.269588 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.269571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.289115 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.289092    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfl5\" (UniqueName: \"kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5\") pod \"kserve-controller-manager-645f4f9567-zx7j5\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") " pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.309129 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.309102    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:56.336142 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.336122    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-lh5w6\""
Jul 20 10:05:56.343242 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.343227    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:05:56.368144 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.368118    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xg8\" (UniqueName: \"kubernetes.io/projected/d4ae3989-afef-4444-86a0-c4f27df04db5-kube-api-access-98xg8\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.368257 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.368193    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4ae3989-afef-4444-86a0-c4f27df04db5-data\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.368514 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.368494    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4ae3989-afef-4444-86a0-c4f27df04db5-data\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.382156 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.382134    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xg8\" (UniqueName: \"kubernetes.io/projected/d4ae3989-afef-4444-86a0-c4f27df04db5-kube-api-access-98xg8\") pod \"seaweedfs-64568bcd49-nml64\" (UID: \"d4ae3989-afef-4444-86a0-c4f27df04db5\") " pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.422978 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.422957    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-89b4t\""
Jul 20 10:05:56.430477 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.430453    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:05:56.494830 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:05:56.494794    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0441d52_237a_4821_a896_2f268fcc13b7.slice/crio-60c12750b75ba54ca41fbcc3ebf6335ce8c6d83db5a2f60c72c9b751f8bb1494 WatchSource:0}: Error finding container 60c12750b75ba54ca41fbcc3ebf6335ce8c6d83db5a2f60c72c9b751f8bb1494: Status 404 returned error can't find the container with id 60c12750b75ba54ca41fbcc3ebf6335ce8c6d83db5a2f60c72c9b751f8bb1494
Jul 20 10:05:56.495261 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.495240    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:05:56.588476 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.588453    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nml64"]
Jul 20 10:05:56.590669 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:05:56.590647    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ae3989_afef_4444_86a0_c4f27df04db5.slice/crio-54173b16b198fb66136f1088c00324415a7d4ab487ef844327fe2e9b54e20498 WatchSource:0}: Error finding container 54173b16b198fb66136f1088c00324415a7d4ab487ef844327fe2e9b54e20498: Status 404 returned error can't find the container with id 54173b16b198fb66136f1088c00324415a7d4ab487ef844327fe2e9b54e20498
Jul 20 10:05:56.945380 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:56.945352    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:57.111259 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:57.111230    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:05:57.364578 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:57.364503    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nml64" event={"ID":"d4ae3989-afef-4444-86a0-c4f27df04db5","Type":"ContainerStarted","Data":"54173b16b198fb66136f1088c00324415a7d4ab487ef844327fe2e9b54e20498"}
Jul 20 10:05:57.365894 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:05:57.365870    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" event={"ID":"a0441d52-237a-4821-a896-2f268fcc13b7","Type":"ContainerStarted","Data":"60c12750b75ba54ca41fbcc3ebf6335ce8c6d83db5a2f60c72c9b751f8bb1494"}
Jul 20 10:06:00.475831 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:00.475812    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 20 10:06:01.389085 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:01.389036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nml64" event={"ID":"d4ae3989-afef-4444-86a0-c4f27df04db5","Type":"ContainerStarted","Data":"f913cf7f6d136bb03cb1455067df2335c0beaad5cd78f034cf66b4d93be762bc"}
Jul 20 10:06:01.389280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:01.389125    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:06:01.390569 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:01.390543    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" event={"ID":"a0441d52-237a-4821-a896-2f268fcc13b7","Type":"ContainerStarted","Data":"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"}
Jul 20 10:06:01.390691 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:01.390594    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:06:01.419229 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:01.419184    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nml64" podStartSLOduration=1.54641553 podStartE2EDuration="5.419170525s" totalImagesPullingTime="3.872754995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:01.416293645 +0000 UTC m=+545.463718390" watchObservedRunningTime="2026-07-20 10:06:01.419170525 +0000 UTC m=+545.466595287"
Jul 20 10:06:07.396952 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:07.396917    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nml64"
Jul 20 10:06:07.419218 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:07.419170    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" podStartSLOduration=8.620389182 podStartE2EDuration="12.419155911s" totalImagesPullingTime="3.798766729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:01.45608594 +0000 UTC m=+545.503510679" watchObservedRunningTime="2026-07-20 10:06:07.419155911 +0000 UTC m=+551.466580656"
Jul 20 10:06:16.451676 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:16.451639    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:32.403619 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:32.403591    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:06:39.846692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.846658    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:06:39.847182 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.846841    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" podUID="a0441d52-237a-4821-a896-2f268fcc13b7" containerName="manager" containerID="cri-o://26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2" gracePeriod=10
Jul 20 10:06:39.869476 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.869416    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-r96hp"]
Jul 20 10:06:39.871910 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.871891    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:39.892769 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.892749    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-r96hp"]
Jul 20 10:06:39.916882 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.916853    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d95521-b827-46c9-aefd-2f6d411d44e2-cert\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:39.917010 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:39.916902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdt8\" (UniqueName: \"kubernetes.io/projected/72d95521-b827-46c9-aefd-2f6d411d44e2-kube-api-access-4fdt8\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.017697 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.017662    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d95521-b827-46c9-aefd-2f6d411d44e2-cert\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.017824 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.017732    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdt8\" (UniqueName: \"kubernetes.io/projected/72d95521-b827-46c9-aefd-2f6d411d44e2-kube-api-access-4fdt8\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.020163 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.020140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d95521-b827-46c9-aefd-2f6d411d44e2-cert\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.027515 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.027487    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdt8\" (UniqueName: \"kubernetes.io/projected/72d95521-b827-46c9-aefd-2f6d411d44e2-kube-api-access-4fdt8\") pod \"kserve-controller-manager-645f4f9567-r96hp\" (UID: \"72d95521-b827-46c9-aefd-2f6d411d44e2\") " pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.085835 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.085815    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:06:40.218813 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.218738    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert\") pod \"a0441d52-237a-4821-a896-2f268fcc13b7\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") "
Jul 20 10:06:40.218953 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.218836    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnfl5\" (UniqueName: \"kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5\") pod \"a0441d52-237a-4821-a896-2f268fcc13b7\" (UID: \"a0441d52-237a-4821-a896-2f268fcc13b7\") "
Jul 20 10:06:40.220843 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.220815    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5" (OuterVolumeSpecName: "kube-api-access-fnfl5") pod "a0441d52-237a-4821-a896-2f268fcc13b7" (UID: "a0441d52-237a-4821-a896-2f268fcc13b7"). InnerVolumeSpecName "kube-api-access-fnfl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:06:40.220905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.220841    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert" (OuterVolumeSpecName: "cert") pod "a0441d52-237a-4821-a896-2f268fcc13b7" (UID: "a0441d52-237a-4821-a896-2f268fcc13b7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:06:40.245023 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.245000    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:40.320322 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.320291    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnfl5\" (UniqueName: \"kubernetes.io/projected/a0441d52-237a-4821-a896-2f268fcc13b7-kube-api-access-fnfl5\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:06:40.320322 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.320324    2546 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0441d52-237a-4821-a896-2f268fcc13b7-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:06:40.373748 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.373723    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-r96hp"]
Jul 20 10:06:40.376002 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:06:40.375968    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d95521_b827_46c9_aefd_2f6d411d44e2.slice/crio-cc05bb591b0fbbcab21def4b637a4e21157f0af7c89914da698b4d07c48079bc WatchSource:0}: Error finding container cc05bb591b0fbbcab21def4b637a4e21157f0af7c89914da698b4d07c48079bc: Status 404 returned error can't find the container with id cc05bb591b0fbbcab21def4b637a4e21157f0af7c89914da698b4d07c48079bc
Jul 20 10:06:40.558324 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.558256    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-r96hp" event={"ID":"72d95521-b827-46c9-aefd-2f6d411d44e2","Type":"ContainerStarted","Data":"cc05bb591b0fbbcab21def4b637a4e21157f0af7c89914da698b4d07c48079bc"}
Jul 20 10:06:40.559513 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.559494    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0441d52-237a-4821-a896-2f268fcc13b7" containerID="26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2" exitCode=0
Jul 20 10:06:40.559606 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.559539    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5"
Jul 20 10:06:40.559606 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.559579    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" event={"ID":"a0441d52-237a-4821-a896-2f268fcc13b7","Type":"ContainerDied","Data":"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"}
Jul 20 10:06:40.559689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.559659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-zx7j5" event={"ID":"a0441d52-237a-4821-a896-2f268fcc13b7","Type":"ContainerDied","Data":"60c12750b75ba54ca41fbcc3ebf6335ce8c6d83db5a2f60c72c9b751f8bb1494"}
Jul 20 10:06:40.559689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.559686    2546 scope.go:117] "RemoveContainer" containerID="26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"
Jul 20 10:06:40.567920 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.567903    2546 scope.go:117] "RemoveContainer" containerID="26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"
Jul 20 10:06:40.568211 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:06:40.568192    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2\": container with ID starting with 26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2 not found: ID does not exist" containerID="26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"
Jul 20 10:06:40.568272 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.568220    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2"} err="failed to get container status \"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2\": rpc error: code = NotFound desc = could not find container \"26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2\": container with ID starting with 26baa439f1cb65276156b464d5ee9419ae84a018a3e057d415aa1ef725cec8e2 not found: ID does not exist"
Jul 20 10:06:40.577517 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.577496    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:06:40.581237 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:40.581219    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-645f4f9567-zx7j5"]
Jul 20 10:06:41.350363 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:41.350334    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:06:41.565352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:41.565315    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-645f4f9567-r96hp" event={"ID":"72d95521-b827-46c9-aefd-2f6d411d44e2","Type":"ContainerStarted","Data":"67e9fbac27928ddd522bbdd6ca1ae6926acc2d674af2d834b07e588e1fbd2062"}
Jul 20 10:06:41.565542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:41.565428    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:06:41.586966 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:41.586923    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-645f4f9567-r96hp" podStartSLOduration=2.210804444 podStartE2EDuration="2.586910078s" totalImagesPullingTime="376.105634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:06:41.58426974 +0000 UTC m=+585.631694495" watchObservedRunningTime="2026-07-20 10:06:41.586910078 +0000 UTC m=+585.634334823"
Jul 20 10:06:42.403286 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:42.403253    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0441d52-237a-4821-a896-2f268fcc13b7" path="/var/lib/kubelet/pods/a0441d52-237a-4821-a896-2f268fcc13b7/volumes"
Jul 20 10:06:56.376111 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:56.376084    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:06:56.377138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:56.377117    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:06:56.379016 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:56.378995    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:06:56.380062 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:06:56.380044    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:07:12.575359 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:12.575330    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-645f4f9567-r96hp"
Jul 20 10:07:13.663639 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.663606    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:13.694522 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.694496    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cq2s7"]
Jul 20 10:07:13.694868 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.694855    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0441d52-237a-4821-a896-2f268fcc13b7" containerName="manager"
Jul 20 10:07:13.694912 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.694870    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0441d52-237a-4821-a896-2f268fcc13b7" containerName="manager"
Jul 20 10:07:13.694951 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.694928    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0441d52-237a-4821-a896-2f268fcc13b7" containerName="manager"
Jul 20 10:07:13.697872 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.697857    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:13.700943 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.700924    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 20 10:07:13.701062 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.701041    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-m2whb\""
Jul 20 10:07:13.710700 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.710680    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cq2s7"]
Jul 20 10:07:13.756082 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.756035    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:13.856026 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.855997    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:13.883706 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.883680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:13.883807 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.883770    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclq7\" (UniqueName: \"kubernetes.io/projected/d3e74f7f-b633-453a-aebe-c55efba6e9cd-kube-api-access-xclq7\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:13.985110 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.985025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:13.985216 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.985161    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclq7\" (UniqueName: \"kubernetes.io/projected/d3e74f7f-b633-453a-aebe-c55efba6e9cd-kube-api-access-xclq7\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:13.985216 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:07:13.985185    2546 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 20 10:07:13.985297 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:07:13.985267    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert podName:d3e74f7f-b633-453a-aebe-c55efba6e9cd nodeName:}" failed. No retries permitted until 2026-07-20 10:07:14.485245319 +0000 UTC m=+618.532670055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert") pod "odh-model-controller-84b54b7f97-cq2s7" (UID: "d3e74f7f-b633-453a-aebe-c55efba6e9cd") : secret "odh-model-controller-webhook-cert" not found
Jul 20 10:07:13.997414 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:13.997388    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclq7\" (UniqueName: \"kubernetes.io/projected/d3e74f7f-b633-453a-aebe-c55efba6e9cd-kube-api-access-xclq7\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:14.182765 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.182730    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.488736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.488708    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:14.491076 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.491045    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3e74f7f-b633-453a-aebe-c55efba6e9cd-cert\") pod \"odh-model-controller-84b54b7f97-cq2s7\" (UID: \"d3e74f7f-b633-453a-aebe-c55efba6e9cd\") " pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:14.498092 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.498047    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.598132 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.598088    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:07:14.607751 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.607735    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:14.737977 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:14.737953    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cq2s7"]
Jul 20 10:07:14.740636 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:07:14.740582    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e74f7f_b633_453a_aebe_c55efba6e9cd.slice/crio-1c82d8fdee719c48c15644b5ac64c759f09085908fc449e325b8487de88d3512 WatchSource:0}: Error finding container 1c82d8fdee719c48c15644b5ac64c759f09085908fc449e325b8487de88d3512: Status 404 returned error can't find the container with id 1c82d8fdee719c48c15644b5ac64c759f09085908fc449e325b8487de88d3512
Jul 20 10:07:15.721812 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:15.721777    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cq2s7" event={"ID":"d3e74f7f-b633-453a-aebe-c55efba6e9cd","Type":"ContainerStarted","Data":"1c82d8fdee719c48c15644b5ac64c759f09085908fc449e325b8487de88d3512"}
Jul 20 10:07:17.733995 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:17.733962    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cq2s7" event={"ID":"d3e74f7f-b633-453a-aebe-c55efba6e9cd","Type":"ContainerStarted","Data":"1538523e4cb40368b22e164203bef314feefae0b9197f0d6ba4fe2def1a103e2"}
Jul 20 10:07:17.734392 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:17.734089    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:07:17.754223 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:17.754182    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-cq2s7" podStartSLOduration=2.353475323 podStartE2EDuration="4.754171119s" totalImagesPullingTime="2.400695796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:07:17.753168875 +0000 UTC m=+621.800593622" watchObservedRunningTime="2026-07-20 10:07:17.754171119 +0000 UTC m=+621.801595919"
Jul 20 10:07:28.741796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:07:28.741769    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-cq2s7"
Jul 20 10:08:48.335812 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:08:48.335774    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:08:49.658191 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:08:49.658108    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.092782 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.092748    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.202547 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.202508    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:09:01.205348 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.205327    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.208171 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208150    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 20 10:09:01.208280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208193    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 20 10:09:01.208280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208205    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 20 10:09:01.208397 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208312    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 20 10:09:01.208451 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208435    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 20 10:09:01.208504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.208478    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 20 10:09:01.209689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.209673    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-npdx9\""
Jul 20 10:09:01.220705 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.220683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:09:01.335721 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.335689    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.335873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.335732    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvrg\" (UniqueName: \"kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.335873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.335760    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.335873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.335784    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.335873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.335817    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.389031 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.388951    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:09:01.391710 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.391688    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.394856 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.394836    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab339-predictor-serving-cert\""
Jul 20 10:09:01.395158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.395143    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ab339-kube-rbac-proxy-sar-config\""
Jul 20 10:09:01.405585 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.405563    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.406861 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.406839    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:09:01.436590 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.436561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.436736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.436604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.436736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.436643    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.436736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.436689    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.436736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.436722    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvrg\" (UniqueName: \"kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.437485 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.437461    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.437616 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.437470    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.437685 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.437620    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.439309 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.439287    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.448607 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.448583    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvrg\" (UniqueName: \"kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg\") pod \"isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.501531 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.501502    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.516051 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.516022    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:01.537778 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.537752    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.537904 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.537804    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd9zh\" (UniqueName: \"kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.537904 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.537894    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.598999 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.598964    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.607832 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.607542    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:09:01.612098 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.611233    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.614369 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.614347    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ab339-predictor-serving-cert\""
Jul 20 10:09:01.614369 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.614359    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ab339-kube-rbac-proxy-sar-config\""
Jul 20 10:09:01.622721 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.622622    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:09:01.638952 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.638928    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.639140 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.638977    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.639140 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.638998    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd9zh\" (UniqueName: \"kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.639266 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:09:01.639204    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ab339-predictor-serving-cert: secret "success-200-isvc-ab339-predictor-serving-cert" not found
Jul 20 10:09:01.639324 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:09:01.639269    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls podName:23772513-4c41-42bb-aa78-c966ae1f0ad5 nodeName:}" failed. No retries permitted until 2026-07-20 10:09:02.139247939 +0000 UTC m=+726.186672667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls") pod "success-200-isvc-ab339-predictor-559b85c779-xld8k" (UID: "23772513-4c41-42bb-aa78-c966ae1f0ad5") : secret "success-200-isvc-ab339-predictor-serving-cert" not found
Jul 20 10:09:01.639648 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.639625    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.647322 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.647300    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd9zh\" (UniqueName: \"kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:01.669120 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.668825    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:09:01.670958 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:09:01.670932    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742f0c97_d4c0_4472_b4c6_5693c3f4c673.slice/crio-55e58699da57420f1ea751d61fe12dfd7c5c0d68bd790156ddabab4232694635 WatchSource:0}: Error finding container 55e58699da57420f1ea751d61fe12dfd7c5c0d68bd790156ddabab4232694635: Status 404 returned error can't find the container with id 55e58699da57420f1ea751d61fe12dfd7c5c0d68bd790156ddabab4232694635
Jul 20 10:09:01.677395 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.677360    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.740261 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.740234    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plbc\" (UniqueName: \"kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.740405 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.740311    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.740405 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.740340    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.748202 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.748179    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:01.841195 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.841167    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plbc\" (UniqueName: \"kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.841299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.841234    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.841299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.841254    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.841407 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:09:01.841369    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ab339-predictor-serving-cert: secret "error-404-isvc-ab339-predictor-serving-cert" not found
Jul 20 10:09:01.841459 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:09:01.841425    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls podName:6633df2b-88a5-450c-875f-19dc0b7f0157 nodeName:}" failed. No retries permitted until 2026-07-20 10:09:02.341407824 +0000 UTC m=+726.388832546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls") pod "error-404-isvc-ab339-predictor-98847c874-24mfc" (UID: "6633df2b-88a5-450c-875f-19dc0b7f0157") : secret "error-404-isvc-ab339-predictor-serving-cert" not found
Jul 20 10:09:01.841943 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.841921    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.849804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.849777    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plbc\" (UniqueName: \"kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:01.913497 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:01.913425    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:02.012345 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.012312    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:09:02.142863 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.142836    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:02.145196 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.145179    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") pod \"success-200-isvc-ab339-predictor-559b85c779-xld8k\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") " pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:02.214439 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.214370    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerStarted","Data":"55e58699da57420f1ea751d61fe12dfd7c5c0d68bd790156ddabab4232694635"}
Jul 20 10:09:02.303374 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.303337    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:02.344855 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.344827    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:02.347156 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.347128    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") pod \"error-404-isvc-ab339-predictor-98847c874-24mfc\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") " pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:02.441817 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.441784    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:09:02.443778 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:09:02.443748    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23772513_4c41_42bb_aa78_c966ae1f0ad5.slice/crio-9a0a1ea4efcb4c0ed34192478a5346125cfb9cabbc4f0fffed3943467f3deb1f WatchSource:0}: Error finding container 9a0a1ea4efcb4c0ed34192478a5346125cfb9cabbc4f0fffed3943467f3deb1f: Status 404 returned error can't find the container with id 9a0a1ea4efcb4c0ed34192478a5346125cfb9cabbc4f0fffed3943467f3deb1f
Jul 20 10:09:02.523857 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.523800    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:02.689497 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:02.689469    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:09:02.691320 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:09:02.691282    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6633df2b_88a5_450c_875f_19dc0b7f0157.slice/crio-5820027053b768cef3e83891c32bd25789039efe2e537eb777e37767d53c37f4 WatchSource:0}: Error finding container 5820027053b768cef3e83891c32bd25789039efe2e537eb777e37767d53c37f4: Status 404 returned error can't find the container with id 5820027053b768cef3e83891c32bd25789039efe2e537eb777e37767d53c37f4
Jul 20 10:09:03.231470 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:03.231384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerStarted","Data":"5820027053b768cef3e83891c32bd25789039efe2e537eb777e37767d53c37f4"}
Jul 20 10:09:03.251463 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:03.251425    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerStarted","Data":"9a0a1ea4efcb4c0ed34192478a5346125cfb9cabbc4f0fffed3943467f3deb1f"}
Jul 20 10:09:20.384356 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:20.384217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerStarted","Data":"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"}
Jul 20 10:09:20.388168 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:20.387574    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerStarted","Data":"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"}
Jul 20 10:09:20.390920 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:20.390842    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerStarted","Data":"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"}
Jul 20 10:09:22.404596 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.404563    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerStarted","Data":"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"}
Jul 20 10:09:22.405209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.404689    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:22.405209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.404736    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:22.406507 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.406475    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:09:22.406630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.406508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerStarted","Data":"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"}
Jul 20 10:09:22.406691 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.406653    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:22.406754 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.406706    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:22.407632 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.407607    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:22.435539 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.435495    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podStartSLOduration=1.948158126 podStartE2EDuration="21.435481949s" totalImagesPullingTime="19.487323823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:22.433624639 +0000 UTC m=+746.481049377" watchObservedRunningTime="2026-07-20 10:09:22.435481949 +0000 UTC m=+746.482906693"
Jul 20 10:09:22.463654 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:22.463604    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podStartSLOduration=2.230956505 podStartE2EDuration="21.463585691s" totalImagesPullingTime="19.232629186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:22.462795778 +0000 UTC m=+746.510220524" watchObservedRunningTime="2026-07-20 10:09:22.463585691 +0000 UTC m=+746.511010436"
Jul 20 10:09:23.414150 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:23.414119    2546 generic.go:356] "Generic (PLEG): container finished" podID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerID="c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344" exitCode=0
Jul 20 10:09:23.414515 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:23.414188    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerDied","Data":"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"}
Jul 20 10:09:23.414601 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:23.414578    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:23.414948 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:23.414909    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:09:28.419924 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:28.419894    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:09:28.420398 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:28.420161    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:09:28.420398 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:28.420372    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:28.420962 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:28.420935    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:09:30.452655 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:30.452617    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerStarted","Data":"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"}
Jul 20 10:09:30.453101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:30.452679    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerStarted","Data":"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"}
Jul 20 10:09:30.453101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:30.452944    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:30.453225 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:30.453109    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:30.454386 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:30.454360    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:09:31.458247 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:31.458204    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:09:36.464434 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:36.464407    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:09:36.465030 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:36.464926    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:09:36.487713 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:36.487666    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podStartSLOduration=9.495077023 podStartE2EDuration="35.487651788s" totalImagesPullingTime="22.219305128s" totalInitContainerRuntime="3.773269637s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:09:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:09:30.478571194 +0000 UTC m=+754.525995950" watchObservedRunningTime="2026-07-20 10:09:36.487651788 +0000 UTC m=+760.535076533"
Jul 20 10:09:38.420732 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:38.420689    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:38.421127 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:38.420892    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:09:46.465102 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:46.465002    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:09:48.420442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:48.420405    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:48.420913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:48.420841    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:09:56.465083 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:56.465026    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:09:58.420350 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:58.420314    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 20 10:09:58.420883 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:09:58.420854    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 20 10:10:06.464917 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:06.464872    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:10:08.421231 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:08.421202    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:10:08.422001 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:08.421982    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:10:16.464852 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:16.464817    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:10:21.118291 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:21.118215    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:21.317092 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:21.317039    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:26.465156 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:26.465122    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:10:31.232503 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.232454    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:31.309588 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.309548    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:10:31.310037 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.310001    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container" containerID="cri-o://e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558" gracePeriod=30
Jul 20 10:10:31.310245 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.310025    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kube-rbac-proxy" containerID="cri-o://c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d" gracePeriod=30
Jul 20 10:10:31.338699 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.338665    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:31.358621 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.358487    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:10:31.359353 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.359006    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container" containerID="cri-o://687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26" gracePeriod=30
Jul 20 10:10:31.359353 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.359231    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kube-rbac-proxy" containerID="cri-o://cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d" gracePeriod=30
Jul 20 10:10:31.407401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.407369    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:31.414484 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.414451    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:10:31.417555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.417538    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.420097 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.420056    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7f819-predictor-serving-cert\""
Jul 20 10:10:31.420202 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.420126    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7f819-kube-rbac-proxy-sar-config\""
Jul 20 10:10:31.428723 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.428704    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:10:31.503482 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.503391    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:31.507151 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.503845    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:10:31.508709 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.508683    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.511716 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.511695    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f819-predictor-serving-cert\""
Jul 20 10:10:31.511946 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.511926    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f819-kube-rbac-proxy-sar-config\""
Jul 20 10:10:31.516274 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.516248    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:10:31.546923 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.546894    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ksw\" (UniqueName: \"kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.547143 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.547124    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.547230 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.547153    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.613832 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.613798    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:31.648106 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxxf\" (UniqueName: \"kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.648221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648129    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.648221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648162    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.648221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648215    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ksw\" (UniqueName: \"kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.648352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648250    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.648352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.648987 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.648964    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.650657 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.650633    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.656933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.656900    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ksw\" (UniqueName: \"kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw\") pod \"success-200-isvc-7f819-predictor-6c79894b85-nc5nt\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") " pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.728244 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.728221    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:31.744039 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.744012    2546 generic.go:356] "Generic (PLEG): container finished" podID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerID="cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d" exitCode=2
Jul 20 10:10:31.744152 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.744100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerDied","Data":"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"}
Jul 20 10:10:31.746039 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.746015    2546 generic.go:356] "Generic (PLEG): container finished" podID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerID="c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d" exitCode=2
Jul 20 10:10:31.746153 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.746089    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerDied","Data":"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"}
Jul 20 10:10:31.748620 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.748596    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxxf\" (UniqueName: \"kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.748717 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.748632    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.748717 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.748681    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.748906 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:31.748832    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7f819-predictor-serving-cert: secret "error-404-isvc-7f819-predictor-serving-cert" not found
Jul 20 10:10:31.749002 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:31.748960    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls podName:5dabde92-fd33-4162-8e56-47e594b50e7a nodeName:}" failed. No retries permitted until 2026-07-20 10:10:32.248916239 +0000 UTC m=+816.296340964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls") pod "error-404-isvc-7f819-predictor-78699b444d-jdwpj" (UID: "5dabde92-fd33-4162-8e56-47e594b50e7a") : secret "error-404-isvc-7f819-predictor-serving-cert" not found
Jul 20 10:10:31.749409 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.749393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.758144 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.758088    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxxf\" (UniqueName: \"kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:31.863160 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:31.863132    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:10:31.866653 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:10:31.866623    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d89d43_57c5_4392_8a8a_9095864d32e8.slice/crio-e34e79b9926771411830c5679af3b6da49138b43bdbc30d96c67f7bc0d774940 WatchSource:0}: Error finding container e34e79b9926771411830c5679af3b6da49138b43bdbc30d96c67f7bc0d774940: Status 404 returned error can't find the container with id e34e79b9926771411830c5679af3b6da49138b43bdbc30d96c67f7bc0d774940
Jul 20 10:10:32.061182 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.061096    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:10:32.251513 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.251481    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:32.253912 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.253887    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") pod \"error-404-isvc-7f819-predictor-78699b444d-jdwpj\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") " pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:32.423205 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.423171    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:32.564212 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.563439    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:10:32.566209 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:10:32.566177    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dabde92_fd33_4162_8e56_47e594b50e7a.slice/crio-3c4f41636ace1df8dfb83ca5245e90786a29049b0081461219d7a126bcb1e32a WatchSource:0}: Error finding container 3c4f41636ace1df8dfb83ca5245e90786a29049b0081461219d7a126bcb1e32a: Status 404 returned error can't find the container with id 3c4f41636ace1df8dfb83ca5245e90786a29049b0081461219d7a126bcb1e32a
Jul 20 10:10:32.753356 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.753324    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerStarted","Data":"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"}
Jul 20 10:10:32.753502 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.753390    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerStarted","Data":"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"}
Jul 20 10:10:32.753502 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.753421    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerStarted","Data":"3c4f41636ace1df8dfb83ca5245e90786a29049b0081461219d7a126bcb1e32a"}
Jul 20 10:10:32.753502 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.753485    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:32.755039 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.755012    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerStarted","Data":"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"}
Jul 20 10:10:32.755144 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.755085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerStarted","Data":"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"}
Jul 20 10:10:32.755144 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.755124    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:32.755221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.755145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerStarted","Data":"e34e79b9926771411830c5679af3b6da49138b43bdbc30d96c67f7bc0d774940"}
Jul 20 10:10:32.776769 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.776718    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podStartSLOduration=1.776704182 podStartE2EDuration="1.776704182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:10:32.775373685 +0000 UTC m=+816.822798454" watchObservedRunningTime="2026-07-20 10:10:32.776704182 +0000 UTC m=+816.824128927"
Jul 20 10:10:32.800852 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:32.800796    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podStartSLOduration=1.800783206 podStartE2EDuration="1.800783206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:10:32.799153484 +0000 UTC m=+816.846578229" watchObservedRunningTime="2026-07-20 10:10:32.800783206 +0000 UTC m=+816.848207950"
Jul 20 10:10:33.414902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.414857    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 20 10:10:33.414902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.414891    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 20 10:10:33.762992 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.762856    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:33.764516 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.763633    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:33.764752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.764724    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:10:33.764925 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:33.764894    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:10:34.669769 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.669744    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:10:34.769780 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.769698    2546 generic.go:356] "Generic (PLEG): container finished" podID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerID="e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558" exitCode=0
Jul 20 10:10:34.769780 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.769744    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"
Jul 20 10:10:34.769954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.769778    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerDied","Data":"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"}
Jul 20 10:10:34.769954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.769840    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k" event={"ID":"23772513-4c41-42bb-aa78-c966ae1f0ad5","Type":"ContainerDied","Data":"9a0a1ea4efcb4c0ed34192478a5346125cfb9cabbc4f0fffed3943467f3deb1f"}
Jul 20 10:10:34.769954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.769867    2546 scope.go:117] "RemoveContainer" containerID="c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"
Jul 20 10:10:34.770396 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.770354    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:10:34.770576 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.770539    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:10:34.774181 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.774164    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") pod \"23772513-4c41-42bb-aa78-c966ae1f0ad5\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") "
Jul 20 10:10:34.774274 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.774231    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd9zh\" (UniqueName: \"kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh\") pod \"23772513-4c41-42bb-aa78-c966ae1f0ad5\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") "
Jul 20 10:10:34.774390 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.774368    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config\") pod \"23772513-4c41-42bb-aa78-c966ae1f0ad5\" (UID: \"23772513-4c41-42bb-aa78-c966ae1f0ad5\") "
Jul 20 10:10:34.774719 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.774693    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23772513-4c41-42bb-aa78-c966ae1f0ad5" (UID: "23772513-4c41-42bb-aa78-c966ae1f0ad5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:10:34.776381 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.776356    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23772513-4c41-42bb-aa78-c966ae1f0ad5" (UID: "23772513-4c41-42bb-aa78-c966ae1f0ad5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:10:34.776689 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.776670    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh" (OuterVolumeSpecName: "kube-api-access-dd9zh") pod "23772513-4c41-42bb-aa78-c966ae1f0ad5" (UID: "23772513-4c41-42bb-aa78-c966ae1f0ad5"). InnerVolumeSpecName "kube-api-access-dd9zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:10:34.779247 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.779231    2546 scope.go:117] "RemoveContainer" containerID="e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"
Jul 20 10:10:34.793287 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.793269    2546 scope.go:117] "RemoveContainer" containerID="c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"
Jul 20 10:10:34.793553 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:34.793537    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d\": container with ID starting with c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d not found: ID does not exist" containerID="c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"
Jul 20 10:10:34.793600 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.793561    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d"} err="failed to get container status \"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d\": rpc error: code = NotFound desc = could not find container \"c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d\": container with ID starting with c41a63911a3e1689f6b8c8b6e3b8b5e52516e911ca59b5b53783e496f183d25d not found: ID does not exist"
Jul 20 10:10:34.793600 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.793581    2546 scope.go:117] "RemoveContainer" containerID="e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"
Jul 20 10:10:34.793819 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:34.793803    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558\": container with ID starting with e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558 not found: ID does not exist" containerID="e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"
Jul 20 10:10:34.793864 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.793826    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558"} err="failed to get container status \"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558\": rpc error: code = NotFound desc = could not find container \"e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558\": container with ID starting with e380d06b1214299d7386c6ef6ffb3b8082dcdfdde25d644afe222a6ed6312558 not found: ID does not exist"
Jul 20 10:10:34.875694 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.875659    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd9zh\" (UniqueName: \"kubernetes.io/projected/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-api-access-dd9zh\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:34.875793 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.875699    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23772513-4c41-42bb-aa78-c966ae1f0ad5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:34.875793 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:34.875723    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23772513-4c41-42bb-aa78-c966ae1f0ad5-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.090961 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.090934    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:10:35.092368 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.092348    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:10:35.097737 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.097716    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ab339-predictor-559b85c779-xld8k"]
Jul 20 10:10:35.178686 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.178663    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") pod \"6633df2b-88a5-450c-875f-19dc0b7f0157\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") "
Jul 20 10:10:35.178811 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.178755    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4plbc\" (UniqueName: \"kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc\") pod \"6633df2b-88a5-450c-875f-19dc0b7f0157\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") "
Jul 20 10:10:35.178811 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.178799    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config\") pod \"6633df2b-88a5-450c-875f-19dc0b7f0157\" (UID: \"6633df2b-88a5-450c-875f-19dc0b7f0157\") "
Jul 20 10:10:35.179233 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.179208    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6633df2b-88a5-450c-875f-19dc0b7f0157" (UID: "6633df2b-88a5-450c-875f-19dc0b7f0157"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:10:35.180678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.180650    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6633df2b-88a5-450c-875f-19dc0b7f0157" (UID: "6633df2b-88a5-450c-875f-19dc0b7f0157"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:10:35.180801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.180733    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc" (OuterVolumeSpecName: "kube-api-access-4plbc") pod "6633df2b-88a5-450c-875f-19dc0b7f0157" (UID: "6633df2b-88a5-450c-875f-19dc0b7f0157"). InnerVolumeSpecName "kube-api-access-4plbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:10:35.279676 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.279616    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6633df2b-88a5-450c-875f-19dc0b7f0157-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.279676 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.279640    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plbc\" (UniqueName: \"kubernetes.io/projected/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-api-access-4plbc\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.279676 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.279650    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6633df2b-88a5-450c-875f-19dc0b7f0157-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:10:35.776492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.776461    2546 generic.go:356] "Generic (PLEG): container finished" podID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerID="687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26" exitCode=0
Jul 20 10:10:35.776921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.776512    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"
Jul 20 10:10:35.776921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.776538    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerDied","Data":"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"}
Jul 20 10:10:35.776921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.776590    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc" event={"ID":"6633df2b-88a5-450c-875f-19dc0b7f0157","Type":"ContainerDied","Data":"5820027053b768cef3e83891c32bd25789039efe2e537eb777e37767d53c37f4"}
Jul 20 10:10:35.776921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.776625    2546 scope.go:117] "RemoveContainer" containerID="cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"
Jul 20 10:10:35.786420 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.786401    2546 scope.go:117] "RemoveContainer" containerID="687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"
Jul 20 10:10:35.794100 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.794061    2546 scope.go:117] "RemoveContainer" containerID="cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"
Jul 20 10:10:35.794340 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:35.794324    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d\": container with ID starting with cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d not found: ID does not exist" containerID="cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"
Jul 20 10:10:35.794409 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.794347    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d"} err="failed to get container status \"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d\": rpc error: code = NotFound desc = could not find container \"cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d\": container with ID starting with cb1ca9d027a8ba2b328328fad3abafff9eadedafa1d8ce5ac3339766f6d0831d not found: ID does not exist"
Jul 20 10:10:35.794409 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.794363    2546 scope.go:117] "RemoveContainer" containerID="687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"
Jul 20 10:10:35.794592 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:10:35.794571    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26\": container with ID starting with 687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26 not found: ID does not exist" containerID="687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"
Jul 20 10:10:35.794634 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.794601    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26"} err="failed to get container status \"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26\": rpc error: code = NotFound desc = could not find container \"687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26\": container with ID starting with 687d88638914737a6ede1a5578fcd75e5bc939f9f49b36e7f7b17f5903fdeb26 not found: ID does not exist"
Jul 20 10:10:35.801080 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.800238    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:10:35.806758 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:35.805197    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ab339-predictor-98847c874-24mfc"]
Jul 20 10:10:36.404296 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:36.404266    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" path="/var/lib/kubelet/pods/23772513-4c41-42bb-aa78-c966ae1f0ad5/volumes"
Jul 20 10:10:36.404694 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:36.404681    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" path="/var/lib/kubelet/pods/6633df2b-88a5-450c-875f-19dc0b7f0157/volumes"
Jul 20 10:10:36.465940 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:36.465900    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 20 10:10:39.775172 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:39.775147    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:10:39.775716 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:39.775494    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:10:39.775716 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:39.775572    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:10:39.775943 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:39.775917    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:10:46.403492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:46.403418    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:10:49.776440 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:49.776400    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:10:49.776813 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:49.776399    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:10:59.776406 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:59.776366    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:10:59.776767 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:10:59.776366    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:11:09.776460 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:09.776418    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:11:09.776841 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:09.776418    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:11:11.095654 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:11.095624    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:11.994757 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:11.994716    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:19.776758 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:19.776728    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:11:19.777198 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:19.776819    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:11:21.211292 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.211262    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.395505 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.395475    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:11:21.395744 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.395707    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container" containerID="cri-o://b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c" gracePeriod=30
Jul 20 10:11:21.395814 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.395751    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kube-rbac-proxy" containerID="cri-o://5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73" gracePeriod=30
Jul 20 10:11:21.407240 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.407212    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.425939 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.425912    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:11:21.426530 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426510    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container"
Jul 20 10:11:21.426530 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426531    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container"
Jul 20 10:11:21.426697 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426549    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kube-rbac-proxy"
Jul 20 10:11:21.426697 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426557    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kube-rbac-proxy"
Jul 20 10:11:21.426798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426700    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container"
Jul 20 10:11:21.426798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426714    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container"
Jul 20 10:11:21.426798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426737    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kube-rbac-proxy"
Jul 20 10:11:21.426798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426746    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kube-rbac-proxy"
Jul 20 10:11:21.426984 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426900    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kube-rbac-proxy"
Jul 20 10:11:21.426984 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426920    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kserve-container"
Jul 20 10:11:21.426984 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426934    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23772513-4c41-42bb-aa78-c966ae1f0ad5" containerName="kserve-container"
Jul 20 10:11:21.426984 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.426943    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6633df2b-88a5-450c-875f-19dc0b7f0157" containerName="kube-rbac-proxy"
Jul 20 10:11:21.431200 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.431180    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.434127 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.434108    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78f99-kube-rbac-proxy-sar-config\""
Jul 20 10:11:21.434250 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.434230    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78f99-predictor-serving-cert\""
Jul 20 10:11:21.446621 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.446600    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:11:21.459223 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.459189    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 20 10:11:21.496597 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.496529    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.556681 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.556652    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:11:21.559715 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.559695    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.562502 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.562482    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78f99-predictor-serving-cert\""
Jul 20 10:11:21.562752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.562734    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78f99-kube-rbac-proxy-sar-config\""
Jul 20 10:11:21.572593 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.572569    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:11:21.573930 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.573902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.574016 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.573959    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722cx\" (UniqueName: \"kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.574106 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.574082    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.596448 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.596425    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.675133 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd2h\" (UniqueName: \"kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.675241 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.675241 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675189    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722cx\" (UniqueName: \"kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.675241 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675231    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.675362 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675252    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.675362 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675271    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.675439 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:11:21.675391    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-78f99-predictor-serving-cert: secret "success-200-isvc-78f99-predictor-serving-cert" not found
Jul 20 10:11:21.675473 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:11:21.675456    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls podName:49cb0401-f5f7-429a-849a-e1f1953bf116 nodeName:}" failed. No retries permitted until 2026-07-20 10:11:22.175434086 +0000 UTC m=+866.222858809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls") pod "success-200-isvc-78f99-predictor-676885594f-6drz2" (UID: "49cb0401-f5f7-429a-849a-e1f1953bf116") : secret "success-200-isvc-78f99-predictor-serving-cert" not found
Jul 20 10:11:21.675857 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.675840    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.689892 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.689864    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-722cx\" (UniqueName: \"kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:21.696602 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.696577    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.776423 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.776343    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.776423 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.776380    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.776423 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.776412    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd2h\" (UniqueName: \"kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.777057 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.777031    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.778792 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.778770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.787039 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.787012    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd2h\" (UniqueName: \"kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h\") pod \"error-404-isvc-78f99-predictor-868689f4b4-fhg95\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") " pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.819315 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.819290    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.871480 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.871454    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:21.924082 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.924021    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:21.997710 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.997681    2546 generic.go:356] "Generic (PLEG): container finished" podID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerID="5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73" exitCode=2
Jul 20 10:11:21.997852 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:21.997733    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerDied","Data":"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"}
Jul 20 10:11:22.004306 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:22.004284    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:11:22.006508 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:11:22.006484    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad7287e_93f2_4ea2_baa7_3fb58160b873.slice/crio-7d09c1dae51ab1683ec097edb311792c6d64d38395203b0ddf491803065a0461 WatchSource:0}: Error finding container 7d09c1dae51ab1683ec097edb311792c6d64d38395203b0ddf491803065a0461: Status 404 returned error can't find the container with id 7d09c1dae51ab1683ec097edb311792c6d64d38395203b0ddf491803065a0461
Jul 20 10:11:22.180334 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:22.180299    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:22.182482 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:22.182457    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") pod \"success-200-isvc-78f99-predictor-676885594f-6drz2\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") " pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:22.342644 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:22.342616    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:22.479133 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:22.479104    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:11:22.482420 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:11:22.482396    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cb0401_f5f7_429a_849a_e1f1953bf116.slice/crio-fb85a8468f62209985c0f1c14a3a618801139f90fef6ac14b2b036653e096fb9 WatchSource:0}: Error finding container fb85a8468f62209985c0f1c14a3a618801139f90fef6ac14b2b036653e096fb9: Status 404 returned error can't find the container with id fb85a8468f62209985c0f1c14a3a618801139f90fef6ac14b2b036653e096fb9
Jul 20 10:11:23.004192 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.004149    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerStarted","Data":"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"}
Jul 20 10:11:23.004401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.004252    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerStarted","Data":"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"}
Jul 20 10:11:23.004401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.004288    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:23.004401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.004318    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerStarted","Data":"7d09c1dae51ab1683ec097edb311792c6d64d38395203b0ddf491803065a0461"}
Jul 20 10:11:23.004401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.004347    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:23.006051 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006019    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:23.006209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006090    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerStarted","Data":"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"}
Jul 20 10:11:23.006209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006147    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerStarted","Data":"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"}
Jul 20 10:11:23.006209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006177    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerStarted","Data":"fb85a8468f62209985c0f1c14a3a618801139f90fef6ac14b2b036653e096fb9"}
Jul 20 10:11:23.006352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006212    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:23.006352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.006245    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:23.007238 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.007213    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:11:23.025032 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.024990    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podStartSLOduration=2.02497845 podStartE2EDuration="2.02497845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:23.02417829 +0000 UTC m=+867.071603036" watchObservedRunningTime="2026-07-20 10:11:23.02497845 +0000 UTC m=+867.072403173"
Jul 20 10:11:23.046401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:23.046363    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podStartSLOduration=2.046350893 podStartE2EDuration="2.046350893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:11:23.045871388 +0000 UTC m=+867.093296134" watchObservedRunningTime="2026-07-20 10:11:23.046350893 +0000 UTC m=+867.093775711"
Jul 20 10:11:24.011489 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:24.011444    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:24.011897 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:24.011540    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:11:25.749335 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.749313    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:11:25.812078 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812006    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmvrg\" (UniqueName: \"kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg\") pod \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") "
Jul 20 10:11:25.812078 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812038    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert\") pod \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") "
Jul 20 10:11:25.812250 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812085    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls\") pod \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") "
Jul 20 10:11:25.812250 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812152    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location\") pod \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") "
Jul 20 10:11:25.812250 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812168    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config\") pod \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\" (UID: \"742f0c97-d4c0-4472-b4c6-5693c3f4c673\") "
Jul 20 10:11:25.812456 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812423    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "742f0c97-d4c0-4472-b4c6-5693c3f4c673" (UID: "742f0c97-d4c0-4472-b4c6-5693c3f4c673"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:11:25.812552 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812484    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "742f0c97-d4c0-4472-b4c6-5693c3f4c673" (UID: "742f0c97-d4c0-4472-b4c6-5693c3f4c673"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 10:11:25.812625 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.812604    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "742f0c97-d4c0-4472-b4c6-5693c3f4c673" (UID: "742f0c97-d4c0-4472-b4c6-5693c3f4c673"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:11:25.814142 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.814117    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "742f0c97-d4c0-4472-b4c6-5693c3f4c673" (UID: "742f0c97-d4c0-4472-b4c6-5693c3f4c673"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:11:25.814223 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.814205    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg" (OuterVolumeSpecName: "kube-api-access-kmvrg") pod "742f0c97-d4c0-4472-b4c6-5693c3f4c673" (UID: "742f0c97-d4c0-4472-b4c6-5693c3f4c673"). InnerVolumeSpecName "kube-api-access-kmvrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:11:25.912804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.912768    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvrg\" (UniqueName: \"kubernetes.io/projected/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-api-access-kmvrg\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:11:25.912804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.912798    2546 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-cabundle-cert\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:11:25.912804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.912812    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/742f0c97-d4c0-4472-b4c6-5693c3f4c673-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:11:25.913024 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.912823    2546 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kserve-provision-location\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:11:25.913024 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:25.912834    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/742f0c97-d4c0-4472-b4c6-5693c3f4c673-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:11:26.022690 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.022658    2546 generic.go:356] "Generic (PLEG): container finished" podID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerID="b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c" exitCode=0
Jul 20 10:11:26.022846 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.022709    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"
Jul 20 10:11:26.022846 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.022741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerDied","Data":"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"}
Jul 20 10:11:26.022846 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.022797    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp" event={"ID":"742f0c97-d4c0-4472-b4c6-5693c3f4c673","Type":"ContainerDied","Data":"55e58699da57420f1ea751d61fe12dfd7c5c0d68bd790156ddabab4232694635"}
Jul 20 10:11:26.022846 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.022834    2546 scope.go:117] "RemoveContainer" containerID="5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"
Jul 20 10:11:26.032934 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.032918    2546 scope.go:117] "RemoveContainer" containerID="b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"
Jul 20 10:11:26.040439 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.040423    2546 scope.go:117] "RemoveContainer" containerID="c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"
Jul 20 10:11:26.048474 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.048453    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:11:26.049438 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.049425    2546 scope.go:117] "RemoveContainer" containerID="5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"
Jul 20 10:11:26.049704 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:11:26.049686    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73\": container with ID starting with 5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73 not found: ID does not exist" containerID="5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"
Jul 20 10:11:26.049749 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.049713    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73"} err="failed to get container status \"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73\": rpc error: code = NotFound desc = could not find container \"5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73\": container with ID starting with 5a1e62da75cd0134dbb793cef24f7a2a3fcd4b4e3f00ab8c886574872be06d73 not found: ID does not exist"
Jul 20 10:11:26.049749 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.049732    2546 scope.go:117] "RemoveContainer" containerID="b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"
Jul 20 10:11:26.049957 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:11:26.049941    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c\": container with ID starting with b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c not found: ID does not exist" containerID="b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"
Jul 20 10:11:26.049995 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.049964    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c"} err="failed to get container status \"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c\": rpc error: code = NotFound desc = could not find container \"b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c\": container with ID starting with b345b79e331562ef468d2a8f1172876620678de605ae285637c05b224a2f080c not found: ID does not exist"
Jul 20 10:11:26.049995 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.049980    2546 scope.go:117] "RemoveContainer" containerID="c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"
Jul 20 10:11:26.050219 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:11:26.050202    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344\": container with ID starting with c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344 not found: ID does not exist" containerID="c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"
Jul 20 10:11:26.050280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.050221    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344"} err="failed to get container status \"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344\": rpc error: code = NotFound desc = could not find container \"c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344\": container with ID starting with c7285d38b3af9f2af9bb37e15773406bfc7b419914bf57dcaafdbc37b2863344 not found: ID does not exist"
Jul 20 10:11:26.054487 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.054467    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-7669b84dcc-69zjp"]
Jul 20 10:11:26.404204 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:26.404172    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" path="/var/lib/kubelet/pods/742f0c97-d4c0-4472-b4c6-5693c3f4c673/volumes"
Jul 20 10:11:29.016416 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:29.016378    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:11:29.016854 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:29.016674    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:11:29.016854 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:29.016833    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:11:29.017247 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:29.017223    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:31.551432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:31.551398    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:31.797431 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:31.797400    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:11:39.017061 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:39.017018    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:11:39.017471 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:39.017190    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:49.017534 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:49.017494    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:11:49.017954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:49.017492    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:56.412774 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:56.412747    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:11:56.413340 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:56.413323    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:11:56.414138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:56.414121    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:11:56.414469 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:56.414454    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:11:59.017583 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:59.017540    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:11:59.017969 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:11:59.017551    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:12:09.017804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:12:09.017777    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:12:09.018222 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:12:09.017846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:12:21.457833 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:12:21.457801    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:12:21.646194 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:12:21.646162    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:04.368244 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:14:04.368204    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:07.272258 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:14:07.272220    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:14:32.696715 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:14:32.696674    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:35.810811 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:15:35.810770    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:15:35.874972 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:15:35.874940    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:28.309997 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:28.309956    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:28.374108 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:28.374060    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:50.765933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:50.765886    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:16:56.454105 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:56.454054    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:16:56.454573 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:56.454403    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:16:56.455212 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:56.455192    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:16:56.455507 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:16:56.455490    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:19:46.387839 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.387758    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:46.638570 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638489    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:19:46.638900 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638885    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kube-rbac-proxy"
Jul 20 10:19:46.638900 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638899    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kube-rbac-proxy"
Jul 20 10:19:46.639022 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638913    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container"
Jul 20 10:19:46.639022 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638921    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container"
Jul 20 10:19:46.639022 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638942    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="storage-initializer"
Jul 20 10:19:46.639022 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.638950    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="storage-initializer"
Jul 20 10:19:46.639249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.639046    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kube-rbac-proxy"
Jul 20 10:19:46.639249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.639059    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="742f0c97-d4c0-4472-b4c6-5693c3f4c673" containerName="kserve-container"
Jul 20 10:19:46.642558 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.642536    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.645757 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.645736    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-242a5-predictor-serving-cert\""
Jul 20 10:19:46.647271 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.647248    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-242a5-kube-rbac-proxy-sar-config\""
Jul 20 10:19:46.666116 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.666094    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:19:46.687437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.687416    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.687550 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.687448    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.687550 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.687500    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbs4k\" (UniqueName: \"kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.703684 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.703657    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:19:46.703894 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.703868    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" containerID="cri-o://b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c" gracePeriod=30
Jul 20 10:19:46.704009 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.703920    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kube-rbac-proxy" containerID="cri-o://49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962" gracePeriod=30
Jul 20 10:19:46.722366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.722336    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:19:46.725951 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.725934    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.728961 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.728937    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-242a5-kube-rbac-proxy-sar-config\""
Jul 20 10:19:46.728961 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.728957    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-242a5-predictor-serving-cert\""
Jul 20 10:19:46.742924 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.742903    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:19:46.787939 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.787896    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.788126 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.787944    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8tx\" (UniqueName: \"kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.788126 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.787998    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.788126 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.788029    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.788126 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.788107    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbs4k\" (UniqueName: \"kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.788366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.788164    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.788796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.788770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.790994 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.790967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.801857 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.801834    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbs4k\" (UniqueName: \"kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k\") pod \"success-200-isvc-242a5-predictor-9b5488df5-wlx2h\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") " pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:46.859615 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.859589    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:19:46.859985 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.859947    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" containerID="cri-o://2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440" gracePeriod=30
Jul 20 10:19:46.860109 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.859972    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kube-rbac-proxy" containerID="cri-o://b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9" gracePeriod=30
Jul 20 10:19:46.888981 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.888932    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8tx\" (UniqueName: \"kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.889125 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.889105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.889202 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.889187    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.889731 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.889712    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.891413 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.891393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.907005 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.906983    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8tx\" (UniqueName: \"kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx\") pod \"error-404-isvc-242a5-predictor-656d487bbb-dhrhq\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") " pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:46.955873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:46.955846    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:47.036325 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.036300    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:47.114366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.114335    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:19:47.114936 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:19:47.114908    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa656b6_0cc5_43db_99d7_e208cbf41d7a.slice/crio-06d0e7293a58c842e968dfc73d2463c360a76674db14879e1150feb5aa8a38bd WatchSource:0}: Error finding container 06d0e7293a58c842e968dfc73d2463c360a76674db14879e1150feb5aa8a38bd: Status 404 returned error can't find the container with id 06d0e7293a58c842e968dfc73d2463c360a76674db14879e1150feb5aa8a38bd
Jul 20 10:19:47.210468 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.210443    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:19:47.217292 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:19:47.217265    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9eef044_e12a_4b03_91f4_06868c830382.slice/crio-0ed4212f1504172ce04c3f363badd66840a7dceaa19457e1e20b36a5d981e163 WatchSource:0}: Error finding container 0ed4212f1504172ce04c3f363badd66840a7dceaa19457e1e20b36a5d981e163: Status 404 returned error can't find the container with id 0ed4212f1504172ce04c3f363badd66840a7dceaa19457e1e20b36a5d981e163
Jul 20 10:19:47.450395 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.450292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerStarted","Data":"d82118e19fe8b2cda76822d85e8261ac8d6042855501f27ec62b31a5b6331f32"}
Jul 20 10:19:47.450395 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.450358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerStarted","Data":"e228ca16f1715bf1ab4ddffc477dc88966305c05eb2031137928c86cf738628b"}
Jul 20 10:19:47.450395 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.450387    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerStarted","Data":"06d0e7293a58c842e968dfc73d2463c360a76674db14879e1150feb5aa8a38bd"}
Jul 20 10:19:47.450933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.450638    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:47.450933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.450684    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:47.452002 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.451974    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:19:47.452251 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.452227    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerStarted","Data":"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"}
Jul 20 10:19:47.452372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.452278    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerStarted","Data":"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"}
Jul 20 10:19:47.452372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.452306    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerStarted","Data":"0ed4212f1504172ce04c3f363badd66840a7dceaa19457e1e20b36a5d981e163"}
Jul 20 10:19:47.452372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.452333    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:47.452372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.452357    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:47.453480 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.453446    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:19:47.454009 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.453991    2546 generic.go:356] "Generic (PLEG): container finished" podID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerID="b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9" exitCode=2
Jul 20 10:19:47.454096 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.454058    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerDied","Data":"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"}
Jul 20 10:19:47.455620 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.455601    2546 generic.go:356] "Generic (PLEG): container finished" podID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerID="49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962" exitCode=2
Jul 20 10:19:47.455701 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.455655    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerDied","Data":"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"}
Jul 20 10:19:47.476625 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.476574    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podStartSLOduration=1.4765609720000001 podStartE2EDuration="1.476560972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:47.475367516 +0000 UTC m=+1371.522792274" watchObservedRunningTime="2026-07-20 10:19:47.476560972 +0000 UTC m=+1371.523985716"
Jul 20 10:19:47.503023 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.502974    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podStartSLOduration=1.502961081 podStartE2EDuration="1.502961081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:19:47.501442091 +0000 UTC m=+1371.548866848" watchObservedRunningTime="2026-07-20 10:19:47.502961081 +0000 UTC m=+1371.550385826"
Jul 20 10:19:47.767795 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:47.767716    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:19:48.463468 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:48.463288    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:19:48.463468 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:48.463384    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:19:49.771106 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:49.771040    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 20 10:19:49.771479 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:49.771037    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 20 10:19:49.775790 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:49.775765    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 20 10:19:49.775945 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:49.775922    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 20 10:19:50.196418 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:19:50.196376    2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d89d43_57c5_4392_8a8a_9095864d32e8.slice/crio-conmon-b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d89d43_57c5_4392_8a8a_9095864d32e8.slice/crio-b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c.scope\": RecentStats: unable to find data in memory cache]"
Jul 20 10:19:50.373000 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.372977    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:19:50.415544 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.415492    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:19:50.423821 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.423803    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config\") pod \"96d89d43-57c5-4392-8a8a-9095864d32e8\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") "
Jul 20 10:19:50.423905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.423863    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls\") pod \"96d89d43-57c5-4392-8a8a-9095864d32e8\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") "
Jul 20 10:19:50.423994 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.423937    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9ksw\" (UniqueName: \"kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw\") pod \"96d89d43-57c5-4392-8a8a-9095864d32e8\" (UID: \"96d89d43-57c5-4392-8a8a-9095864d32e8\") "
Jul 20 10:19:50.424195 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.424173    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96d89d43-57c5-4392-8a8a-9095864d32e8" (UID: "96d89d43-57c5-4392-8a8a-9095864d32e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:19:50.424280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.424249    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.425864 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.425842    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96d89d43-57c5-4392-8a8a-9095864d32e8" (UID: "96d89d43-57c5-4392-8a8a-9095864d32e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:19:50.425949 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.425885    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw" (OuterVolumeSpecName: "kube-api-access-j9ksw") pod "96d89d43-57c5-4392-8a8a-9095864d32e8" (UID: "96d89d43-57c5-4392-8a8a-9095864d32e8"). InnerVolumeSpecName "kube-api-access-j9ksw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:19:50.480972 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.480938    2546 generic.go:356] "Generic (PLEG): container finished" podID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerID="2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440" exitCode=0
Jul 20 10:19:50.481155 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.481007    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"
Jul 20 10:19:50.481155 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.481018    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerDied","Data":"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"}
Jul 20 10:19:50.481155 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.481119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj" event={"ID":"5dabde92-fd33-4162-8e56-47e594b50e7a","Type":"ContainerDied","Data":"3c4f41636ace1df8dfb83ca5245e90786a29049b0081461219d7a126bcb1e32a"}
Jul 20 10:19:50.481322 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.481157    2546 scope.go:117] "RemoveContainer" containerID="b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"
Jul 20 10:19:50.482717 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.482694    2546 generic.go:356] "Generic (PLEG): container finished" podID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerID="b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c" exitCode=0
Jul 20 10:19:50.482825 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.482744    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"
Jul 20 10:19:50.482825 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.482770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerDied","Data":"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"}
Jul 20 10:19:50.482825 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.482823    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt" event={"ID":"96d89d43-57c5-4392-8a8a-9095864d32e8","Type":"ContainerDied","Data":"e34e79b9926771411830c5679af3b6da49138b43bdbc30d96c67f7bc0d774940"}
Jul 20 10:19:50.489877 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.489847    2546 scope.go:117] "RemoveContainer" containerID="2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"
Jul 20 10:19:50.497743 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.497726    2546 scope.go:117] "RemoveContainer" containerID="b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"
Jul 20 10:19:50.497974 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:19:50.497955    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9\": container with ID starting with b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9 not found: ID does not exist" containerID="b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"
Jul 20 10:19:50.498044 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.497986    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9"} err="failed to get container status \"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9\": rpc error: code = NotFound desc = could not find container \"b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9\": container with ID starting with b7b369961b549615bb7ba8e868ea7b3e5b92b9a29606eb865a8a5f4bb67438d9 not found: ID does not exist"
Jul 20 10:19:50.498044 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.498010    2546 scope.go:117] "RemoveContainer" containerID="2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"
Jul 20 10:19:50.498273 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:19:50.498257    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440\": container with ID starting with 2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440 not found: ID does not exist" containerID="2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"
Jul 20 10:19:50.498344 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.498280    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440"} err="failed to get container status \"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440\": rpc error: code = NotFound desc = could not find container \"2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440\": container with ID starting with 2effd1c950dfe78cd23d590113f9da72f7c2eca271182bece574b3da4d2ff440 not found: ID does not exist"
Jul 20 10:19:50.498344 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.498301    2546 scope.go:117] "RemoveContainer" containerID="49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"
Jul 20 10:19:50.505850 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.505769    2546 scope.go:117] "RemoveContainer" containerID="b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"
Jul 20 10:19:50.506346 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.506325    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:19:50.510749 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.510730    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7f819-predictor-6c79894b85-nc5nt"]
Jul 20 10:19:50.513307 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.513292    2546 scope.go:117] "RemoveContainer" containerID="49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"
Jul 20 10:19:50.513546 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:19:50.513528    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962\": container with ID starting with 49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962 not found: ID does not exist" containerID="49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"
Jul 20 10:19:50.513596 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.513551    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962"} err="failed to get container status \"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962\": rpc error: code = NotFound desc = could not find container \"49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962\": container with ID starting with 49f08824d112e36ade0c08e79b7220aa00b5803df2447a3758891e75f4b01962 not found: ID does not exist"
Jul 20 10:19:50.513596 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.513566    2546 scope.go:117] "RemoveContainer" containerID="b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"
Jul 20 10:19:50.513794 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:19:50.513777    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c\": container with ID starting with b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c not found: ID does not exist" containerID="b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"
Jul 20 10:19:50.513839 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.513800    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c"} err="failed to get container status \"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c\": rpc error: code = NotFound desc = could not find container \"b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c\": container with ID starting with b8362a9aa0ae623b9ae26dd7db88160dc32fe7cf54912560663b3d227583f34c not found: ID does not exist"
Jul 20 10:19:50.525101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525084    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config\") pod \"5dabde92-fd33-4162-8e56-47e594b50e7a\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") "
Jul 20 10:19:50.525179 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525123    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bxxf\" (UniqueName: \"kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf\") pod \"5dabde92-fd33-4162-8e56-47e594b50e7a\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") "
Jul 20 10:19:50.525219 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") pod \"5dabde92-fd33-4162-8e56-47e594b50e7a\" (UID: \"5dabde92-fd33-4162-8e56-47e594b50e7a\") "
Jul 20 10:19:50.525410 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525398    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ksw\" (UniqueName: \"kubernetes.io/projected/96d89d43-57c5-4392-8a8a-9095864d32e8-kube-api-access-j9ksw\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.525444 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525413    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d89d43-57c5-4392-8a8a-9095864d32e8-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.525444 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.525429    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5dabde92-fd33-4162-8e56-47e594b50e7a" (UID: "5dabde92-fd33-4162-8e56-47e594b50e7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:19:50.527033 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.527012    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf" (OuterVolumeSpecName: "kube-api-access-7bxxf") pod "5dabde92-fd33-4162-8e56-47e594b50e7a" (UID: "5dabde92-fd33-4162-8e56-47e594b50e7a"). InnerVolumeSpecName "kube-api-access-7bxxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:19:50.527033 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.527014    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5dabde92-fd33-4162-8e56-47e594b50e7a" (UID: "5dabde92-fd33-4162-8e56-47e594b50e7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:19:50.626707 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.626688    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5dabde92-fd33-4162-8e56-47e594b50e7a-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.626707 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.626708    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.626843 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.626717    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bxxf\" (UniqueName: \"kubernetes.io/projected/5dabde92-fd33-4162-8e56-47e594b50e7a-kube-api-access-7bxxf\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:19:50.804191 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.804163    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:19:50.810137 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:50.810116    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f819-predictor-78699b444d-jdwpj"]
Jul 20 10:19:52.403478 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:52.403446    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" path="/var/lib/kubelet/pods/5dabde92-fd33-4162-8e56-47e594b50e7a/volumes"
Jul 20 10:19:52.403864 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:52.403850    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" path="/var/lib/kubelet/pods/96d89d43-57c5-4392-8a8a-9095864d32e8/volumes"
Jul 20 10:19:53.468395 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:53.468368    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:19:53.468824 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:53.468792    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:19:53.468901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:53.468832    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:19:53.469167 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:19:53.469140    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:20:03.468884 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:03.468832    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:20:03.469302 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:03.469176    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:20:13.469639 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:13.469592    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:20:13.470012 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:13.469593    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:20:13.562156 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:13.562123    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:23.468907 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:23.468868    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 20 10:20:23.469392 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:23.469173    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 20 10:20:33.469195 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:33.469166    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:20:33.469975 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:33.469956    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:20:36.060369 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.060336    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:36.135614 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.135583    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:20:36.135847 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.135813    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" containerID="cri-o://8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f" gracePeriod=30
Jul 20 10:20:36.135945 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.135857    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kube-rbac-proxy" containerID="cri-o://bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44" gracePeriod=30
Jul 20 10:20:36.171023 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.170959    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:36.178996 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.178951    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:20:36.179352 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.179250    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" containerID="cri-o://795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157" gracePeriod=30
Jul 20 10:20:36.179491 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.179402    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kube-rbac-proxy" containerID="cri-o://582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba" gracePeriod=30
Jul 20 10:20:36.229176 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229146    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:20:36.229630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229612    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kube-rbac-proxy"
Jul 20 10:20:36.229677 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229633    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kube-rbac-proxy"
Jul 20 10:20:36.229677 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229654    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kube-rbac-proxy"
Jul 20 10:20:36.229677 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229659    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kube-rbac-proxy"
Jul 20 10:20:36.229677 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229673    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229679    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229686    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229692    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229761    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229772    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dabde92-fd33-4162-8e56-47e594b50e7a" containerName="kube-rbac-proxy"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229778    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kserve-container"
Jul 20 10:20:36.229801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.229784    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d89d43-57c5-4392-8a8a-9095864d32e8" containerName="kube-rbac-proxy"
Jul 20 10:20:36.233335 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.233316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.235913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.235895    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-24b48-predictor-serving-cert\""
Jul 20 10:20:36.236011 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.235900    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-24b48-kube-rbac-proxy-sar-config\""
Jul 20 10:20:36.245321 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.245300    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:20:36.265643 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.265621    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:36.318122 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.318037    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5mk\" (UniqueName: \"kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.318224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.318147    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.318224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.318183    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.331955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.331933    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:20:36.335839 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.335811    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:36.337517 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.337487    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.340560 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.340541    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-24b48-predictor-serving-cert\""
Jul 20 10:20:36.340671 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.340559    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-24b48-kube-rbac-proxy-sar-config\""
Jul 20 10:20:36.347982 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.347959    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:20:36.419369 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419333    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.419541 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419408    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsvbf\" (UniqueName: \"kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.419541 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419438    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.419672 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419535    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.419672 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419579    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5mk\" (UniqueName: \"kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.419774 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.419827 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:36.419805    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-24b48-predictor-serving-cert: secret "success-200-isvc-24b48-predictor-serving-cert" not found
Jul 20 10:20:36.419893 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:36.419883    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls podName:6326a41c-0694-4a0d-992d-69854cc8802b nodeName:}" failed. No retries permitted until 2026-07-20 10:20:36.919863137 +0000 UTC m=+1420.967287861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls") pod "success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" (UID: "6326a41c-0694-4a0d-992d-69854cc8802b") : secret "success-200-isvc-24b48-predictor-serving-cert" not found
Jul 20 10:20:36.420003 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.419980    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.431241 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.431221    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5mk\" (UniqueName: \"kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.469801 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.469764    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:36.520282 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.520255    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.520407 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.520353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsvbf\" (UniqueName: \"kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.520407 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.520372    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.520527 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:36.520421    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-24b48-predictor-serving-cert: secret "error-404-isvc-24b48-predictor-serving-cert" not found
Jul 20 10:20:36.520527 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:36.520502    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls podName:67383b80-8bd0-4351-b085-1ed83f615709 nodeName:}" failed. No retries permitted until 2026-07-20 10:20:37.020481535 +0000 UTC m=+1421.067906275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls") pod "error-404-isvc-24b48-predictor-74c9c85698-qg4h5" (UID: "67383b80-8bd0-4351-b085-1ed83f615709") : secret "error-404-isvc-24b48-predictor-serving-cert" not found
Jul 20 10:20:36.520997 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.520978    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.529581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.529554    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsvbf\" (UniqueName: \"kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:36.711365 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.711338    2546 generic.go:356] "Generic (PLEG): container finished" podID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerID="582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba" exitCode=2
Jul 20 10:20:36.711524 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.711411    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerDied","Data":"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"}
Jul 20 10:20:36.713081 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.713045    2546 generic.go:356] "Generic (PLEG): container finished" podID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerID="bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44" exitCode=2
Jul 20 10:20:36.713177 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.713093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerDied","Data":"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"}
Jul 20 10:20:36.924343 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.924310    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:36.926758 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:36.926741    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") pod \"success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") " pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:37.005704 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.005630    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:37.025718 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.025695    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:37.027935 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.027916    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") pod \"error-404-isvc-24b48-predictor-74c9c85698-qg4h5\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") " pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:37.144012 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.143986    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:37.250572 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.250539    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:37.285362 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.285328    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:20:37.286196 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:20:37.286166    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6326a41c_0694_4a0d_992d_69854cc8802b.slice/crio-777642a359f4fa098ec0f763acbe961cce862afe1320e2f9832036fd52adab54 WatchSource:0}: Error finding container 777642a359f4fa098ec0f763acbe961cce862afe1320e2f9832036fd52adab54: Status 404 returned error can't find the container with id 777642a359f4fa098ec0f763acbe961cce862afe1320e2f9832036fd52adab54
Jul 20 10:20:37.404137 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.404112    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:20:37.408245 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:20:37.408221    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67383b80_8bd0_4351_b085_1ed83f615709.slice/crio-61c8cfad5d57486efd17e63545d0b3e53d235d3da5a476940840c2901462b02c WatchSource:0}: Error finding container 61c8cfad5d57486efd17e63545d0b3e53d235d3da5a476940840c2901462b02c: Status 404 returned error can't find the container with id 61c8cfad5d57486efd17e63545d0b3e53d235d3da5a476940840c2901462b02c
Jul 20 10:20:37.719575 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.719538    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerStarted","Data":"ed41361d547ba14d61ef51dbbdc48bbf7875a2abe1d4f74e90858ff7c88147b9"}
Jul 20 10:20:37.719779 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.719605    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerStarted","Data":"a66377637205b09cc5a6c773c1e70039b7200a263b30c828dc4cf48a2a6f297b"}
Jul 20 10:20:37.719779 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.719637    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerStarted","Data":"61c8cfad5d57486efd17e63545d0b3e53d235d3da5a476940840c2901462b02c"}
Jul 20 10:20:37.719779 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.719669    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:37.719779 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.719700    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:37.721236 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721204    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:20:37.721566 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721544    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerStarted","Data":"9f6aa2169502f8e72af729636ce751036870088a9afe85af138051a3c462984e"}
Jul 20 10:20:37.721669 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721603    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:37.721669 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721637    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:37.721788 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721667    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerStarted","Data":"0cf8679617d872763cb8d0563e7a2f4356ddf24239114e115741a44b29f4c5e1"}
Jul 20 10:20:37.721788 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.721697    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerStarted","Data":"777642a359f4fa098ec0f763acbe961cce862afe1320e2f9832036fd52adab54"}
Jul 20 10:20:37.722542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.722520    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:20:37.738936 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.738893    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podStartSLOduration=1.738879493 podStartE2EDuration="1.738879493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:37.737402405 +0000 UTC m=+1421.784827157" watchObservedRunningTime="2026-07-20 10:20:37.738879493 +0000 UTC m=+1421.786304238"
Jul 20 10:20:37.756326 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:37.756284    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podStartSLOduration=1.756267566 podStartE2EDuration="1.756267566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:37.754992114 +0000 UTC m=+1421.802416870" watchObservedRunningTime="2026-07-20 10:20:37.756267566 +0000 UTC m=+1421.803692314"
Jul 20 10:20:38.726288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:38.725928    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:20:38.726288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:38.726126    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:20:39.012630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.012537    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 20 10:20:39.012796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.012543    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 20 10:20:39.017263 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.017236    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 20 10:20:39.017377 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.017239    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 20 10:20:39.232192 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.232168    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:20:39.345750 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.345722    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config\") pod \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") "
Jul 20 10:20:39.345905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.345796    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls\") pod \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") "
Jul 20 10:20:39.345905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.345840    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcd2h\" (UniqueName: \"kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h\") pod \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\" (UID: \"6ad7287e-93f2-4ea2-baa7-3fb58160b873\") "
Jul 20 10:20:39.346097 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.346054    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ad7287e-93f2-4ea2-baa7-3fb58160b873" (UID: "6ad7287e-93f2-4ea2-baa7-3fb58160b873"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:39.347920 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.347891    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ad7287e-93f2-4ea2-baa7-3fb58160b873" (UID: "6ad7287e-93f2-4ea2-baa7-3fb58160b873"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:39.348057 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.347943    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h" (OuterVolumeSpecName: "kube-api-access-jcd2h") pod "6ad7287e-93f2-4ea2-baa7-3fb58160b873" (UID: "6ad7287e-93f2-4ea2-baa7-3fb58160b873"). InnerVolumeSpecName "kube-api-access-jcd2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:39.447319 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.447292    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcd2h\" (UniqueName: \"kubernetes.io/projected/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-api-access-jcd2h\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:39.447319 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.447314    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ad7287e-93f2-4ea2-baa7-3fb58160b873-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:39.447319 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.447324    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ad7287e-93f2-4ea2-baa7-3fb58160b873-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:39.679767 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.679744    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:20:39.731913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.731879    2546 generic.go:356] "Generic (PLEG): container finished" podID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerID="795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157" exitCode=0
Jul 20 10:20:39.732342 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.731928    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"
Jul 20 10:20:39.732342 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.731954    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerDied","Data":"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"}
Jul 20 10:20:39.732342 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.732027    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95" event={"ID":"6ad7287e-93f2-4ea2-baa7-3fb58160b873","Type":"ContainerDied","Data":"7d09c1dae51ab1683ec097edb311792c6d64d38395203b0ddf491803065a0461"}
Jul 20 10:20:39.732342 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.732083    2546 scope.go:117] "RemoveContainer" containerID="582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"
Jul 20 10:20:39.733712 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.733688    2546 generic.go:356] "Generic (PLEG): container finished" podID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerID="8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f" exitCode=0
Jul 20 10:20:39.733819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.733728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerDied","Data":"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"}
Jul 20 10:20:39.733819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.733754    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"
Jul 20 10:20:39.733819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.733781    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2" event={"ID":"49cb0401-f5f7-429a-849a-e1f1953bf116","Type":"ContainerDied","Data":"fb85a8468f62209985c0f1c14a3a618801139f90fef6ac14b2b036653e096fb9"}
Jul 20 10:20:39.741870 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.741856    2546 scope.go:117] "RemoveContainer" containerID="795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"
Jul 20 10:20:39.750298 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.750278    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config\") pod \"49cb0401-f5f7-429a-849a-e1f1953bf116\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") "
Jul 20 10:20:39.750390 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.750306    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") pod \"49cb0401-f5f7-429a-849a-e1f1953bf116\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") "
Jul 20 10:20:39.750390 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.750345    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-722cx\" (UniqueName: \"kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx\") pod \"49cb0401-f5f7-429a-849a-e1f1953bf116\" (UID: \"49cb0401-f5f7-429a-849a-e1f1953bf116\") "
Jul 20 10:20:39.750710 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.750683    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49cb0401-f5f7-429a-849a-e1f1953bf116" (UID: "49cb0401-f5f7-429a-849a-e1f1953bf116"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:20:39.751688 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.751671    2546 scope.go:117] "RemoveContainer" containerID="582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"
Jul 20 10:20:39.752520 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:39.752493    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba\": container with ID starting with 582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba not found: ID does not exist" containerID="582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"
Jul 20 10:20:39.752598 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752530    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba"} err="failed to get container status \"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba\": rpc error: code = NotFound desc = could not find container \"582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba\": container with ID starting with 582af176273d223c2959870f789aff4dea64384a6a6da70c89f0bb0131e272ba not found: ID does not exist"
Jul 20 10:20:39.752598 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752552    2546 scope.go:117] "RemoveContainer" containerID="795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"
Jul 20 10:20:39.752735 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752577    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx" (OuterVolumeSpecName: "kube-api-access-722cx") pod "49cb0401-f5f7-429a-849a-e1f1953bf116" (UID: "49cb0401-f5f7-429a-849a-e1f1953bf116"). InnerVolumeSpecName "kube-api-access-722cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:20:39.752735 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752599    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49cb0401-f5f7-429a-849a-e1f1953bf116" (UID: "49cb0401-f5f7-429a-849a-e1f1953bf116"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:20:39.752816 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:39.752795    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157\": container with ID starting with 795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157 not found: ID does not exist" containerID="795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"
Jul 20 10:20:39.752854 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752816    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157"} err="failed to get container status \"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157\": rpc error: code = NotFound desc = could not find container \"795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157\": container with ID starting with 795283a5984500ca7bc913904f54824372fe23d1dde64cc9549d33e09317f157 not found: ID does not exist"
Jul 20 10:20:39.752854 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.752832    2546 scope.go:117] "RemoveContainer" containerID="bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"
Jul 20 10:20:39.754667 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.754647    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:20:39.759060 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.759037    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78f99-predictor-868689f4b4-fhg95"]
Jul 20 10:20:39.760994 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.760968    2546 scope.go:117] "RemoveContainer" containerID="8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"
Jul 20 10:20:39.768533 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.768518    2546 scope.go:117] "RemoveContainer" containerID="bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"
Jul 20 10:20:39.768754 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:39.768737    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44\": container with ID starting with bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44 not found: ID does not exist" containerID="bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"
Jul 20 10:20:39.768821 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.768760    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44"} err="failed to get container status \"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44\": rpc error: code = NotFound desc = could not find container \"bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44\": container with ID starting with bd24ba22f5b8c35b1097dc8c9f2affca0b6d3b324866cac5e5d0ab33e3239e44 not found: ID does not exist"
Jul 20 10:20:39.768821 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.768779    2546 scope.go:117] "RemoveContainer" containerID="8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"
Jul 20 10:20:39.769010 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:20:39.768996    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f\": container with ID starting with 8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f not found: ID does not exist" containerID="8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"
Jul 20 10:20:39.769056 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.769013    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f"} err="failed to get container status \"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f\": rpc error: code = NotFound desc = could not find container \"8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f\": container with ID starting with 8e0822baa1a229337a6b4ddf3851d28ecc5bf867ef4be8374b7cb9e70f8b9a9f not found: ID does not exist"
Jul 20 10:20:39.851717 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.851695    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722cx\" (UniqueName: \"kubernetes.io/projected/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-api-access-722cx\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:39.851798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.851719    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49cb0401-f5f7-429a-849a-e1f1953bf116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:39.851798 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:39.851732    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49cb0401-f5f7-429a-849a-e1f1953bf116-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:20:40.057007 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:40.056959    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:20:40.061507 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:40.061484    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78f99-predictor-676885594f-6drz2"]
Jul 20 10:20:40.405424 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:40.405392    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" path="/var/lib/kubelet/pods/49cb0401-f5f7-429a-849a-e1f1953bf116/volumes"
Jul 20 10:20:40.405792 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:40.405779    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" path="/var/lib/kubelet/pods/6ad7287e-93f2-4ea2-baa7-3fb58160b873/volumes"
Jul 20 10:20:43.729828 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:43.729784    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:20:43.730299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:43.730149    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:20:43.730299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:43.730224    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:20:43.730594 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:43.730568    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:20:46.553254 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:46.553219    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:47.060268 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:47.060236    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:53.730721 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:53.730597    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:20:53.731501 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:53.730811    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:20:53.776879 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:53.776853    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.650567 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.650534    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.716784 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.716749    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:20:56.717100 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.717048    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container" containerID="cri-o://d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11" gracePeriod=30
Jul 20 10:20:56.717349 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.717061    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kube-rbac-proxy" containerID="cri-o://9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3" gracePeriod=30
Jul 20 10:20:56.749785 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.749753    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.766298 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.766276    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:20:56.766521 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.766493    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container" containerID="cri-o://e228ca16f1715bf1ab4ddffc477dc88966305c05eb2031137928c86cf738628b" gracePeriod=30
Jul 20 10:20:56.766632 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.766514    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kube-rbac-proxy" containerID="cri-o://d82118e19fe8b2cda76822d85e8261ac8d6042855501f27ec62b31a5b6331f32" gracePeriod=30
Jul 20 10:20:56.812767 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.812742    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:20:56.813219 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813193    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813219 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813209    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813225    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813235    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813247    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813256    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813277    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813282    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813346    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kserve-container"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813360    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813366    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cb0401-f5f7-429a-849a-e1f1953bf116" containerName="kube-rbac-proxy"
Jul 20 10:20:56.813367 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.813372    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad7287e-93f2-4ea2-baa7-3fb58160b873" containerName="kserve-container"
Jul 20 10:20:56.818466 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.818445    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:56.821276 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.821250    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53302-kube-rbac-proxy-sar-config\""
Jul 20 10:20:56.821385 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.821250    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-53302-predictor-serving-cert\""
Jul 20 10:20:56.830930 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.830907    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:20:56.832358 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.832325    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.913647 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.913621    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.915093 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.915059    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:20:56.918707 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.918685    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:56.921370 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.921351    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53302-predictor-serving-cert\""
Jul 20 10:20:56.921462 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.921355    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53302-kube-rbac-proxy-sar-config\""
Jul 20 10:20:56.929906 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.929888    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:20:56.980309 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.980276    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:56.994404 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.994377    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:56.994549 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.994467    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:56.994549 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:56.994485    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq8g\" (UniqueName: \"kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.081280 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.081211    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:20:57.095417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095391    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.095504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095426    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbn5\" (UniqueName: \"kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.095504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095469    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.095504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095499    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.095663 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095538    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.095663 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.095561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq8g\" (UniqueName: \"kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.096100 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.096052    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.097869 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.097852    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.105123 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.105101    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq8g\" (UniqueName: \"kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g\") pod \"success-200-isvc-53302-predictor-687d4d7db9-w9qtn\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") " pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.129938 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.129918    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.196872 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.196667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbn5\" (UniqueName: \"kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.196872 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.196743    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.196872 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.196789    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.197719 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.197692    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.199304 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.199282    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.205950 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.205929    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbn5\" (UniqueName: \"kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5\") pod \"error-404-isvc-53302-predictor-5b4fdf4964-w5cwv\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") " pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.230537 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.230507    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.265707 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.265675    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:20:57.268307 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:20:57.268279    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d1388b_807e_4cbe_836f_b594315998e8.slice/crio-89bec5524a65a1d75845ed2a7b335ae06c46005048de6c531adf9eb38c84c33a WatchSource:0}: Error finding container 89bec5524a65a1d75845ed2a7b335ae06c46005048de6c531adf9eb38c84c33a: Status 404 returned error can't find the container with id 89bec5524a65a1d75845ed2a7b335ae06c46005048de6c531adf9eb38c84c33a
Jul 20 10:20:57.374292 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.374270    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:20:57.375499 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:20:57.375475    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0648a8_63bf_4a92_8017_6921e4d12346.slice/crio-15e30ca8caac04603f15a5a483d1f27d0b7ec7dd9ec706e310c5e892690a0d78 WatchSource:0}: Error finding container 15e30ca8caac04603f15a5a483d1f27d0b7ec7dd9ec706e310c5e892690a0d78: Status 404 returned error can't find the container with id 15e30ca8caac04603f15a5a483d1f27d0b7ec7dd9ec706e310c5e892690a0d78
Jul 20 10:20:57.824406 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.824315    2546 generic.go:356] "Generic (PLEG): container finished" podID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerID="d82118e19fe8b2cda76822d85e8261ac8d6042855501f27ec62b31a5b6331f32" exitCode=2
Jul 20 10:20:57.824406 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.824391    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerDied","Data":"d82118e19fe8b2cda76822d85e8261ac8d6042855501f27ec62b31a5b6331f32"}
Jul 20 10:20:57.825937 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.825914    2546 generic.go:356] "Generic (PLEG): container finished" podID="b9eef044-e12a-4b03-91f4-06868c830382" containerID="9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3" exitCode=2
Jul 20 10:20:57.826056 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.825986    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerDied","Data":"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"}
Jul 20 10:20:57.827683 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.827659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerStarted","Data":"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"}
Jul 20 10:20:57.827808 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.827706    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerStarted","Data":"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"}
Jul 20 10:20:57.827878 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.827817    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.827933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.827893    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:20:57.827933 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.827924    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerStarted","Data":"89bec5524a65a1d75845ed2a7b335ae06c46005048de6c531adf9eb38c84c33a"}
Jul 20 10:20:57.828927 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.828899    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:20:57.829443 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.829422    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerStarted","Data":"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"}
Jul 20 10:20:57.829516 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.829464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerStarted","Data":"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"}
Jul 20 10:20:57.829516 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.829495    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerStarted","Data":"15e30ca8caac04603f15a5a483d1f27d0b7ec7dd9ec706e310c5e892690a0d78"}
Jul 20 10:20:57.829581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.829517    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.829612 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.829580    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:20:57.830607 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.830585    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:20:57.847097 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.847038    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podStartSLOduration=1.847026761 podStartE2EDuration="1.847026761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:57.846057518 +0000 UTC m=+1441.893482262" watchObservedRunningTime="2026-07-20 10:20:57.847026761 +0000 UTC m=+1441.894451506"
Jul 20 10:20:57.864819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:57.864762    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podStartSLOduration=1.8647435639999999 podStartE2EDuration="1.864743564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:20:57.864396778 +0000 UTC m=+1441.911821523" watchObservedRunningTime="2026-07-20 10:20:57.864743564 +0000 UTC m=+1441.912168311"
Jul 20 10:20:58.464479 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:58.464441    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 20 10:20:58.464939 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:58.464441    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 20 10:20:58.834550 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:58.834515    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:20:58.834550 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:58.834542    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:20:59.843012 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:59.842979    2546 generic.go:356] "Generic (PLEG): container finished" podID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerID="e228ca16f1715bf1ab4ddffc477dc88966305c05eb2031137928c86cf738628b" exitCode=0
Jul 20 10:20:59.843366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:59.843042    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerDied","Data":"e228ca16f1715bf1ab4ddffc477dc88966305c05eb2031137928c86cf738628b"}
Jul 20 10:20:59.922275 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:20:59.922251    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:21:00.022745 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.022667    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbs4k\" (UniqueName: \"kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k\") pod \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") "
Jul 20 10:21:00.022745 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.022722    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config\") pod \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") "
Jul 20 10:21:00.022914 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.022821    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls\") pod \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\" (UID: \"2aa656b6-0cc5-43db-99d7-e208cbf41d7a\") "
Jul 20 10:21:00.023103 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.023049    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2aa656b6-0cc5-43db-99d7-e208cbf41d7a" (UID: "2aa656b6-0cc5-43db-99d7-e208cbf41d7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:21:00.023229 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.023144    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.024760 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.024729    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2aa656b6-0cc5-43db-99d7-e208cbf41d7a" (UID: "2aa656b6-0cc5-43db-99d7-e208cbf41d7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:21:00.024855 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.024816    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k" (OuterVolumeSpecName: "kube-api-access-pbs4k") pod "2aa656b6-0cc5-43db-99d7-e208cbf41d7a" (UID: "2aa656b6-0cc5-43db-99d7-e208cbf41d7a"). InnerVolumeSpecName "kube-api-access-pbs4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:21:00.123947 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.123919    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbs4k\" (UniqueName: \"kubernetes.io/projected/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-kube-api-access-pbs4k\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.123947 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.123944    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2aa656b6-0cc5-43db-99d7-e208cbf41d7a-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.467504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.467474    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:21:00.628543 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.628513    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls\") pod \"b9eef044-e12a-4b03-91f4-06868c830382\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") "
Jul 20 10:21:00.628694 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.628571    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s8tx\" (UniqueName: \"kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx\") pod \"b9eef044-e12a-4b03-91f4-06868c830382\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") "
Jul 20 10:21:00.628694 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.628617    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config\") pod \"b9eef044-e12a-4b03-91f4-06868c830382\" (UID: \"b9eef044-e12a-4b03-91f4-06868c830382\") "
Jul 20 10:21:00.628989 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.628966    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9eef044-e12a-4b03-91f4-06868c830382" (UID: "b9eef044-e12a-4b03-91f4-06868c830382"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:21:00.630569 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.630543    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx" (OuterVolumeSpecName: "kube-api-access-4s8tx") pod "b9eef044-e12a-4b03-91f4-06868c830382" (UID: "b9eef044-e12a-4b03-91f4-06868c830382"). InnerVolumeSpecName "kube-api-access-4s8tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:21:00.630569 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.630549    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9eef044-e12a-4b03-91f4-06868c830382" (UID: "b9eef044-e12a-4b03-91f4-06868c830382"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:21:00.730186 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.730158    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9eef044-e12a-4b03-91f4-06868c830382-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.730186 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.730188    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8tx\" (UniqueName: \"kubernetes.io/projected/b9eef044-e12a-4b03-91f4-06868c830382-kube-api-access-4s8tx\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.730345 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.730203    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9eef044-e12a-4b03-91f4-06868c830382-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:21:00.852422 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.852396    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"
Jul 20 10:21:00.852810 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.852430    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h" event={"ID":"2aa656b6-0cc5-43db-99d7-e208cbf41d7a","Type":"ContainerDied","Data":"06d0e7293a58c842e968dfc73d2463c360a76674db14879e1150feb5aa8a38bd"}
Jul 20 10:21:00.852810 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.852507    2546 scope.go:117] "RemoveContainer" containerID="d82118e19fe8b2cda76822d85e8261ac8d6042855501f27ec62b31a5b6331f32"
Jul 20 10:21:00.853970 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.853948    2546 generic.go:356] "Generic (PLEG): container finished" podID="b9eef044-e12a-4b03-91f4-06868c830382" containerID="d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11" exitCode=0
Jul 20 10:21:00.854089 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.853995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerDied","Data":"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"}
Jul 20 10:21:00.854089 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.854007    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"
Jul 20 10:21:00.854089 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.854034    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq" event={"ID":"b9eef044-e12a-4b03-91f4-06868c830382","Type":"ContainerDied","Data":"0ed4212f1504172ce04c3f363badd66840a7dceaa19457e1e20b36a5d981e163"}
Jul 20 10:21:00.862706 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.862642    2546 scope.go:117] "RemoveContainer" containerID="e228ca16f1715bf1ab4ddffc477dc88966305c05eb2031137928c86cf738628b"
Jul 20 10:21:00.872010 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.871984    2546 scope.go:117] "RemoveContainer" containerID="9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"
Jul 20 10:21:00.880593 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.880575    2546 scope.go:117] "RemoveContainer" containerID="d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"
Jul 20 10:21:00.881319 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.881297    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:21:00.883104 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.883083    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-242a5-predictor-9b5488df5-wlx2h"]
Jul 20 10:21:00.887664 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.887649    2546 scope.go:117] "RemoveContainer" containerID="9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"
Jul 20 10:21:00.887907 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:21:00.887889    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3\": container with ID starting with 9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3 not found: ID does not exist" containerID="9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"
Jul 20 10:21:00.887952 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.887915    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3"} err="failed to get container status \"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3\": rpc error: code = NotFound desc = could not find container \"9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3\": container with ID starting with 9672bc3956e1e09bd9e18f0a8cd8b8c94c74e107f0ecbce10c74dbe21b4365d3 not found: ID does not exist"
Jul 20 10:21:00.887952 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.887931    2546 scope.go:117] "RemoveContainer" containerID="d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"
Jul 20 10:21:00.888179 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:21:00.888164    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11\": container with ID starting with d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11 not found: ID does not exist" containerID="d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"
Jul 20 10:21:00.888230 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.888185    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11"} err="failed to get container status \"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11\": rpc error: code = NotFound desc = could not find container \"d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11\": container with ID starting with d11215765f4ff3605cf4c84bbc08100914871a2f17c5f02c3db4a46471c8ba11 not found: ID does not exist"
Jul 20 10:21:00.894171 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.894150    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:21:00.898511 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:00.898491    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-242a5-predictor-656d487bbb-dhrhq"]
Jul 20 10:21:02.403640 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:02.403604    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" path="/var/lib/kubelet/pods/2aa656b6-0cc5-43db-99d7-e208cbf41d7a/volumes"
Jul 20 10:21:02.404027 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:02.404012    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9eef044-e12a-4b03-91f4-06868c830382" path="/var/lib/kubelet/pods/b9eef044-e12a-4b03-91f4-06868c830382/volumes"
Jul 20 10:21:03.731062 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.731026    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:21:03.731062 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.731052    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:21:03.839155 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.839130    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:21:03.839436 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.839418    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:21:03.839658 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.839629    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:21:03.839806 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:03.839782    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:21:13.730279 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:13.730238    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:21:13.730723 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:13.730525    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:21:13.839737 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:13.839701    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:21:13.839904 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:13.839695    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:21:23.730869 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:23.730828    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 20 10:21:23.730869 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:23.730855    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 20 10:21:23.839879 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:23.839840    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:21:23.840024 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:23.839855    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:21:33.731245 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:33.731207    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:21:33.731731 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:33.731315    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:21:33.840432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:33.840390    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:21:33.840609 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:33.840449    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:21:43.840221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:43.840191    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:21:43.840643 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:43.840286    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:21:46.274681 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:46.274624    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:46.414975 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:46.414947    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.351490 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.351447    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.437564 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.437534    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:21:56.437853 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.437824    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container" containerID="cri-o://0cf8679617d872763cb8d0563e7a2f4356ddf24239114e115741a44b29f4c5e1" gracePeriod=30
Jul 20 10:21:56.437932 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.437902    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kube-rbac-proxy" containerID="cri-o://9f6aa2169502f8e72af729636ce751036870088a9afe85af138051a3c462984e" gracePeriod=30
Jul 20 10:21:56.463726 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.463698    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.481162 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.481116    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:21:56.481429 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.481401    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container" containerID="cri-o://a66377637205b09cc5a6c773c1e70039b7200a263b30c828dc4cf48a2a6f297b" gracePeriod=30
Jul 20 10:21:56.481665 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.481525    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kube-rbac-proxy" containerID="cri-o://ed41361d547ba14d61ef51dbbdc48bbf7875a2abe1d4f74e90858ff7c88147b9" gracePeriod=30
Jul 20 10:21:56.494100 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.494024    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:21:56.495001 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.494981    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:21:56.495536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.495516    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:21:56.496388 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.496371    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:21:56.534408 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534383    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:21:56.534828 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534811    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container"
Jul 20 10:21:56.534901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534829    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container"
Jul 20 10:21:56.534901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534849    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kube-rbac-proxy"
Jul 20 10:21:56.534901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534858    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kube-rbac-proxy"
Jul 20 10:21:56.534901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534877    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kube-rbac-proxy"
Jul 20 10:21:56.534901 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534885    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kube-rbac-proxy"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534931    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.534941    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.535101    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kserve-container"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.535115    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9eef044-e12a-4b03-91f4-06868c830382" containerName="kube-rbac-proxy"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.535127    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kube-rbac-proxy"
Jul 20 10:21:56.535170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.535141    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa656b6-0cc5-43db-99d7-e208cbf41d7a" containerName="kserve-container"
Jul 20 10:21:56.538599 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.538580    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.541488 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.541469    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-64e78-kube-rbac-proxy-sar-config\""
Jul 20 10:21:56.542112 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.542092    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-64e78-predictor-serving-cert\""
Jul 20 10:21:56.549506 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.549472    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.551744 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.551727    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:21:56.626259 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.626231    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.631896 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.631873    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:21:56.635408 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.635393    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.638325 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.638309    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-64e78-predictor-serving-cert\""
Jul 20 10:21:56.638867 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.638842    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-64e78-kube-rbac-proxy-sar-config\""
Jul 20 10:21:56.645119 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.645098    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:21:56.698772 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.698742    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95tk\" (UniqueName: \"kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.698913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.698802    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.698913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.698850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.792346 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.792274    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.799416 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799392    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.799561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799428    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.799561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799457    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.799561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799476    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.799703 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95tk\" (UniqueName: \"kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.799703 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:21:56.799571    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-64e78-predictor-serving-cert: secret "success-200-isvc-64e78-predictor-serving-cert" not found
Jul 20 10:21:56.799703 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.799588    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvb2j\" (UniqueName: \"kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.799986 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:21:56.799966    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls podName:50e96327-c06e-4abc-b3b7-4c3992a3a34f nodeName:}" failed. No retries permitted until 2026-07-20 10:21:57.299614362 +0000 UTC m=+1501.347039087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls") pod "success-200-isvc-64e78-predictor-676cf96d7f-fsxql" (UID: "50e96327-c06e-4abc-b3b7-4c3992a3a34f") : secret "success-200-isvc-64e78-predictor-serving-cert" not found
Jul 20 10:21:56.800530 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.800507    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.814886 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.814863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95tk\" (UniqueName: \"kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:56.893299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.893261    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:56.900417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.900385    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.900561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.900427    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.900561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.900511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvb2j\" (UniqueName: \"kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.901220 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.901197    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.902912 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.902893    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.908641 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.908620    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvb2j\" (UniqueName: \"kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j\") pod \"error-404-isvc-64e78-predictor-b57c5f466-7vqfm\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") " pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.946206 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.946180    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:56.997099 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:56.997020    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:57.078543 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.078519    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:21:57.080610 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:21:57.080577    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfbf234_83ae_48f9_bfbe_cefac206efc7.slice/crio-8a9bc668d09d94e54f2e1e8ba3fb90729757db1d2011942d27d81bd1e2a4f972 WatchSource:0}: Error finding container 8a9bc668d09d94e54f2e1e8ba3fb90729757db1d2011942d27d81bd1e2a4f972: Status 404 returned error can't find the container with id 8a9bc668d09d94e54f2e1e8ba3fb90729757db1d2011942d27d81bd1e2a4f972
Jul 20 10:21:57.129054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.129031    2546 generic.go:356] "Generic (PLEG): container finished" podID="6326a41c-0694-4a0d-992d-69854cc8802b" containerID="9f6aa2169502f8e72af729636ce751036870088a9afe85af138051a3c462984e" exitCode=2
Jul 20 10:21:57.129170 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.129107    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerDied","Data":"9f6aa2169502f8e72af729636ce751036870088a9afe85af138051a3c462984e"}
Jul 20 10:21:57.130200 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.130177    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerStarted","Data":"8a9bc668d09d94e54f2e1e8ba3fb90729757db1d2011942d27d81bd1e2a4f972"}
Jul 20 10:21:57.132044 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.132022    2546 generic.go:356] "Generic (PLEG): container finished" podID="67383b80-8bd0-4351-b085-1ed83f615709" containerID="ed41361d547ba14d61ef51dbbdc48bbf7875a2abe1d4f74e90858ff7c88147b9" exitCode=2
Jul 20 10:21:57.132168 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.132093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerDied","Data":"ed41361d547ba14d61ef51dbbdc48bbf7875a2abe1d4f74e90858ff7c88147b9"}
Jul 20 10:21:57.304491 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.304458    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:57.306778 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.306759    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") pod \"success-200-isvc-64e78-predictor-676cf96d7f-fsxql\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") " pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:57.330015 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.329971    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:21:57.451089 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.451033    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:57.582824 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:57.582654    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:21:57.585049 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:21:57.585026    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e96327_c06e_4abc_b3b7_4c3992a3a34f.slice/crio-93f3e89d8a457aed0e895936bf17b3187e1f4a73abdb0c2aae4a57ae72664807 WatchSource:0}: Error finding container 93f3e89d8a457aed0e895936bf17b3187e1f4a73abdb0c2aae4a57ae72664807: Status 404 returned error can't find the container with id 93f3e89d8a457aed0e895936bf17b3187e1f4a73abdb0c2aae4a57ae72664807
Jul 20 10:21:58.138648 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.138611    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerStarted","Data":"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"}
Jul 20 10:21:58.138862 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.138681    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerStarted","Data":"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"}
Jul 20 10:21:58.138862 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.138726    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:58.138862 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.138754    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:21:58.140191 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.140158    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:21:58.140526 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.140508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerStarted","Data":"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"}
Jul 20 10:21:58.140587 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.140551    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerStarted","Data":"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"}
Jul 20 10:21:58.140587 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.140573    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerStarted","Data":"93f3e89d8a457aed0e895936bf17b3187e1f4a73abdb0c2aae4a57ae72664807"}
Jul 20 10:21:58.140664 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.140595    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:58.159555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.159515    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podStartSLOduration=2.15949901 podStartE2EDuration="2.15949901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:58.157988451 +0000 UTC m=+1502.205413217" watchObservedRunningTime="2026-07-20 10:21:58.15949901 +0000 UTC m=+1502.206923755"
Jul 20 10:21:58.178143 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.178098    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podStartSLOduration=2.178083524 podStartE2EDuration="2.178083524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:21:58.176675246 +0000 UTC m=+1502.224099990" watchObservedRunningTime="2026-07-20 10:21:58.178083524 +0000 UTC m=+1502.225508266"
Jul 20 10:21:58.726157 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.726112    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 20 10:21:58.726157 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:58.726124    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 20 10:21:59.145444 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:59.145405    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:21:59.145659 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:59.145480    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:21:59.146467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:21:59.146442    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:00.153181 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.153152    2546 generic.go:356] "Generic (PLEG): container finished" podID="67383b80-8bd0-4351-b085-1ed83f615709" containerID="a66377637205b09cc5a6c773c1e70039b7200a263b30c828dc4cf48a2a6f297b" exitCode=0
Jul 20 10:22:00.153562 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.153227    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerDied","Data":"a66377637205b09cc5a6c773c1e70039b7200a263b30c828dc4cf48a2a6f297b"}
Jul 20 10:22:00.156193 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.156113    2546 generic.go:356] "Generic (PLEG): container finished" podID="6326a41c-0694-4a0d-992d-69854cc8802b" containerID="0cf8679617d872763cb8d0563e7a2f4356ddf24239114e115741a44b29f4c5e1" exitCode=0
Jul 20 10:22:00.156622 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.156191    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerDied","Data":"0cf8679617d872763cb8d0563e7a2f4356ddf24239114e115741a44b29f4c5e1"}
Jul 20 10:22:00.156703 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.156652    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:00.199745 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.199725    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:22:00.242526 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.242503    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:22:00.331246 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.331215    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") pod \"6326a41c-0694-4a0d-992d-69854cc8802b\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") "
Jul 20 10:22:00.331403 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.331307    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q5mk\" (UniqueName: \"kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk\") pod \"6326a41c-0694-4a0d-992d-69854cc8802b\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") "
Jul 20 10:22:00.331403 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.331347    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config\") pod \"6326a41c-0694-4a0d-992d-69854cc8802b\" (UID: \"6326a41c-0694-4a0d-992d-69854cc8802b\") "
Jul 20 10:22:00.331758 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.331723    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6326a41c-0694-4a0d-992d-69854cc8802b" (UID: "6326a41c-0694-4a0d-992d-69854cc8802b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:00.331893 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.331811    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6326a41c-0694-4a0d-992d-69854cc8802b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:00.333380 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.333360    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6326a41c-0694-4a0d-992d-69854cc8802b" (UID: "6326a41c-0694-4a0d-992d-69854cc8802b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:00.333467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.333419    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk" (OuterVolumeSpecName: "kube-api-access-5q5mk") pod "6326a41c-0694-4a0d-992d-69854cc8802b" (UID: "6326a41c-0694-4a0d-992d-69854cc8802b"). InnerVolumeSpecName "kube-api-access-5q5mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:00.432468 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432443    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsvbf\" (UniqueName: \"kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf\") pod \"67383b80-8bd0-4351-b085-1ed83f615709\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") "
Jul 20 10:22:00.432581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432480    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") pod \"67383b80-8bd0-4351-b085-1ed83f615709\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") "
Jul 20 10:22:00.432581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432550    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config\") pod \"67383b80-8bd0-4351-b085-1ed83f615709\" (UID: \"67383b80-8bd0-4351-b085-1ed83f615709\") "
Jul 20 10:22:00.432765 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432749    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6326a41c-0694-4a0d-992d-69854cc8802b-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:00.432804 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432775    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5mk\" (UniqueName: \"kubernetes.io/projected/6326a41c-0694-4a0d-992d-69854cc8802b-kube-api-access-5q5mk\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:00.432922 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.432905    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67383b80-8bd0-4351-b085-1ed83f615709" (UID: "67383b80-8bd0-4351-b085-1ed83f615709"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:22:00.434510 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.434486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67383b80-8bd0-4351-b085-1ed83f615709" (UID: "67383b80-8bd0-4351-b085-1ed83f615709"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:22:00.434510 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.434501    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf" (OuterVolumeSpecName: "kube-api-access-fsvbf") pod "67383b80-8bd0-4351-b085-1ed83f615709" (UID: "67383b80-8bd0-4351-b085-1ed83f615709"). InnerVolumeSpecName "kube-api-access-fsvbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:22:00.534138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.534055    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67383b80-8bd0-4351-b085-1ed83f615709-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:00.534138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.534098    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsvbf\" (UniqueName: \"kubernetes.io/projected/67383b80-8bd0-4351-b085-1ed83f615709-kube-api-access-fsvbf\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:00.534138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:00.534108    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67383b80-8bd0-4351-b085-1ed83f615709-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:22:01.163829 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.163800    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"
Jul 20 10:22:01.163829 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.163806    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5" event={"ID":"67383b80-8bd0-4351-b085-1ed83f615709","Type":"ContainerDied","Data":"61c8cfad5d57486efd17e63545d0b3e53d235d3da5a476940840c2901462b02c"}
Jul 20 10:22:01.164372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.163879    2546 scope.go:117] "RemoveContainer" containerID="ed41361d547ba14d61ef51dbbdc48bbf7875a2abe1d4f74e90858ff7c88147b9"
Jul 20 10:22:01.166273 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.166243    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf" event={"ID":"6326a41c-0694-4a0d-992d-69854cc8802b","Type":"ContainerDied","Data":"777642a359f4fa098ec0f763acbe961cce862afe1320e2f9832036fd52adab54"}
Jul 20 10:22:01.166369 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.166272    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"
Jul 20 10:22:01.178477 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.178458    2546 scope.go:117] "RemoveContainer" containerID="a66377637205b09cc5a6c773c1e70039b7200a263b30c828dc4cf48a2a6f297b"
Jul 20 10:22:01.188783 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.188764    2546 scope.go:117] "RemoveContainer" containerID="9f6aa2169502f8e72af729636ce751036870088a9afe85af138051a3c462984e"
Jul 20 10:22:01.188988 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.188969    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:22:01.193541 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.193514    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-24b48-predictor-7d8fb7d579-wz4wf"]
Jul 20 10:22:01.196858 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.196843    2546 scope.go:117] "RemoveContainer" containerID="0cf8679617d872763cb8d0563e7a2f4356ddf24239114e115741a44b29f4c5e1"
Jul 20 10:22:01.205471 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.205448    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:22:01.207835 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:01.207816    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-24b48-predictor-74c9c85698-qg4h5"]
Jul 20 10:22:02.405115 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:02.405061    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" path="/var/lib/kubelet/pods/6326a41c-0694-4a0d-992d-69854cc8802b/volumes"
Jul 20 10:22:02.405556 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:02.405539    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67383b80-8bd0-4351-b085-1ed83f615709" path="/var/lib/kubelet/pods/67383b80-8bd0-4351-b085-1ed83f615709/volumes"
Jul 20 10:22:04.156466 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:04.156438    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:22:04.156951 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:04.156926    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:22:05.162742 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:05.162696    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:22:05.163366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:05.163208    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:14.157681 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:14.157642    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:22:15.163876 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:15.163828    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:24.157897 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:24.157804    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:22:25.163272 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:25.163226    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:34.157465 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:34.157410    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:22:35.163343 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:35.163296    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 20 10:22:44.157823 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:44.157788    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:22:45.163809 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:45.163782    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:22:57.336516 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:57.336475    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:22:57.427701 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:22:57.427660    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:26:19.595951 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:19.595907    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:26:45.356561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:45.356524    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:26:56.529154 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:56.529117    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:26:56.530333 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:56.530311    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:26:56.531315 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:56.531294    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:26:56.532440 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:26:56.532419    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:30:11.578254 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.578218    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:11.716915 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.716878    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:11.772609 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.772570    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:30:11.773162 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773142    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kube-rbac-proxy"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773165    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kube-rbac-proxy"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773187    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773195    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773204    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kube-rbac-proxy"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773213    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kube-rbac-proxy"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773238    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container"
Jul 20 10:30:11.773249 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773248    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container"
Jul 20 10:30:11.773492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773339    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kube-rbac-proxy"
Jul 20 10:30:11.773492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773354    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kserve-container"
Jul 20 10:30:11.773492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773365    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6326a41c-0694-4a0d-992d-69854cc8802b" containerName="kserve-container"
Jul 20 10:30:11.773492 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.773373    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="67383b80-8bd0-4351-b085-1ed83f615709" containerName="kube-rbac-proxy"
Jul 20 10:30:11.776983 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.776959    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.781251 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.781230    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a956e-predictor-serving-cert\""
Jul 20 10:30:11.781529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.781509    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a956e-kube-rbac-proxy-sar-config\""
Jul 20 10:30:11.808104 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.808062    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:30:11.825946 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.825914    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:30:11.826143 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.826121    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" containerID="cri-o://0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7" gracePeriod=30
Jul 20 10:30:11.826231 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.826195    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kube-rbac-proxy" containerID="cri-o://b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee" gracePeriod=30
Jul 20 10:30:11.849267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.849235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.849377 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.849298    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdppz\" (UniqueName: \"kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.849452 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.849379    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.859843 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.859821    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:11.878654 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.878631    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:30:11.882442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.882426    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:11.886183 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.886162    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a956e-kube-rbac-proxy-sar-config\""
Jul 20 10:30:11.886539 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.886523    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a956e-predictor-serving-cert\""
Jul 20 10:30:11.898831 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.898812    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:30:11.950060 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950033    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:11.950262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950107    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.950262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950139    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbxl\" (UniqueName: \"kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:11.950262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950167    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:11.950262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950229    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdppz\" (UniqueName: \"kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.950744 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950282    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.950932 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.950899    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.953241 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.953218    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.967400 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.967379    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdppz\" (UniqueName: \"kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz\") pod \"success-200-isvc-a956e-predictor-6f7cf8c9-8n552\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") " pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:11.972843 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.972818    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:30:11.973057 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.973031    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" containerID="cri-o://57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda" gracePeriod=30
Jul 20 10:30:11.973160 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.973077    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kube-rbac-proxy" containerID="cri-o://4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714" gracePeriod=30
Jul 20 10:30:11.978438 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:11.978151    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:12.050831 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.050797    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.050987 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.050847    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbxl\" (UniqueName: \"kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.050987 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.050874    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.051605 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.051556    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.053528 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.053500    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.053906 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.053881    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:30:12.059954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.059932    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbxl\" (UniqueName: \"kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl\") pod \"error-404-isvc-a956e-predictor-78896946fd-x7gsj\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") " pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.089345 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.089327    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:12.192441 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.192407    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.225153 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.225127    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:30:12.227602 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:30:12.227575    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002f075b_ee91_4ce2_b135_2704a2312877.slice/crio-d9c30b21b60045e1985c69a72be10cfaace85845195fc76380d52ff4511d81cc WatchSource:0}: Error finding container d9c30b21b60045e1985c69a72be10cfaace85845195fc76380d52ff4511d81cc: Status 404 returned error can't find the container with id d9c30b21b60045e1985c69a72be10cfaace85845195fc76380d52ff4511d81cc
Jul 20 10:30:12.336909 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.336885    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:30:12.338662 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:30:12.338636    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ee06bf_5653_4a01_9147_c0adf85c66a5.slice/crio-60e13478c7fca9884cdc40dafff963e3c7758b539bbb97dfa79d5bbaf3ebf98d WatchSource:0}: Error finding container 60e13478c7fca9884cdc40dafff963e3c7758b539bbb97dfa79d5bbaf3ebf98d: Status 404 returned error can't find the container with id 60e13478c7fca9884cdc40dafff963e3c7758b539bbb97dfa79d5bbaf3ebf98d
Jul 20 10:30:12.514736 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.514703    2546 generic.go:356] "Generic (PLEG): container finished" podID="22d1388b-807e-4cbe-836f-b594315998e8" containerID="b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee" exitCode=2
Jul 20 10:30:12.514902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.514776    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerDied","Data":"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"}
Jul 20 10:30:12.516403 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.516380    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerStarted","Data":"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"}
Jul 20 10:30:12.516740 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.516437    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.516740 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.516460    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:12.516740 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.516481    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerStarted","Data":"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"}
Jul 20 10:30:12.516740 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.516508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerStarted","Data":"60e13478c7fca9884cdc40dafff963e3c7758b539bbb97dfa79d5bbaf3ebf98d"}
Jul 20 10:30:12.518151 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.518122    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:12.518259 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.518226    2546 generic.go:356] "Generic (PLEG): container finished" podID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerID="4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714" exitCode=2
Jul 20 10:30:12.518313 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.518252    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerDied","Data":"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"}
Jul 20 10:30:12.519889 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.519867    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerStarted","Data":"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"}
Jul 20 10:30:12.519982 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.519915    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerStarted","Data":"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"}
Jul 20 10:30:12.519982 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.519968    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerStarted","Data":"d9c30b21b60045e1985c69a72be10cfaace85845195fc76380d52ff4511d81cc"}
Jul 20 10:30:12.520054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.519996    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:12.538082 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.538034    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podStartSLOduration=1.5380223210000001 podStartE2EDuration="1.538022321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:30:12.536108309 +0000 UTC m=+1996.583533057" watchObservedRunningTime="2026-07-20 10:30:12.538022321 +0000 UTC m=+1996.585447065"
Jul 20 10:30:12.556916 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:12.556870    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podStartSLOduration=1.556854978 podStartE2EDuration="1.556854978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:30:12.554443989 +0000 UTC m=+1996.601868735" watchObservedRunningTime="2026-07-20 10:30:12.556854978 +0000 UTC m=+1996.604279725"
Jul 20 10:30:13.526011 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.525977    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:13.526426 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.526080    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:13.527262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.527238    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:13.835158 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.835123    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 20 10:30:13.835316 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.835137    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 20 10:30:13.840581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.840562    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 20 10:30:13.840630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:13.840597    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 20 10:30:14.531250 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:14.531214    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:14.986252 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:14.986224    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:30:15.026927 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.026906    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:30:15.081301 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081268    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config\") pod \"4c0648a8-63bf-4a92-8017-6921e4d12346\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") "
Jul 20 10:30:15.081432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081306    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsq8g\" (UniqueName: \"kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g\") pod \"22d1388b-807e-4cbe-836f-b594315998e8\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") "
Jul 20 10:30:15.081432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081330    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config\") pod \"22d1388b-807e-4cbe-836f-b594315998e8\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") "
Jul 20 10:30:15.081432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081366    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls\") pod \"4c0648a8-63bf-4a92-8017-6921e4d12346\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") "
Jul 20 10:30:15.081432 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081424    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls\") pod \"22d1388b-807e-4cbe-836f-b594315998e8\" (UID: \"22d1388b-807e-4cbe-836f-b594315998e8\") "
Jul 20 10:30:15.081592 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081460    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lbn5\" (UniqueName: \"kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5\") pod \"4c0648a8-63bf-4a92-8017-6921e4d12346\" (UID: \"4c0648a8-63bf-4a92-8017-6921e4d12346\") "
Jul 20 10:30:15.081696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.081665    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22d1388b-807e-4cbe-836f-b594315998e8" (UID: "22d1388b-807e-4cbe-836f-b594315998e8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:30:15.082093 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.082043    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c0648a8-63bf-4a92-8017-6921e4d12346" (UID: "4c0648a8-63bf-4a92-8017-6921e4d12346"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:30:15.083479 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.083455    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c0648a8-63bf-4a92-8017-6921e4d12346" (UID: "4c0648a8-63bf-4a92-8017-6921e4d12346"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:30:15.083550 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.083508    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22d1388b-807e-4cbe-836f-b594315998e8" (UID: "22d1388b-807e-4cbe-836f-b594315998e8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:30:15.083595 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.083546    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5" (OuterVolumeSpecName: "kube-api-access-5lbn5") pod "4c0648a8-63bf-4a92-8017-6921e4d12346" (UID: "4c0648a8-63bf-4a92-8017-6921e4d12346"). InnerVolumeSpecName "kube-api-access-5lbn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:30:15.083595 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.083552    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g" (OuterVolumeSpecName: "kube-api-access-lsq8g") pod "22d1388b-807e-4cbe-836f-b594315998e8" (UID: "22d1388b-807e-4cbe-836f-b594315998e8"). InnerVolumeSpecName "kube-api-access-lsq8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182447    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182470    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsq8g\" (UniqueName: \"kubernetes.io/projected/22d1388b-807e-4cbe-836f-b594315998e8-kube-api-access-lsq8g\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182481    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22d1388b-807e-4cbe-836f-b594315998e8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182491    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c0648a8-63bf-4a92-8017-6921e4d12346-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182500    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22d1388b-807e-4cbe-836f-b594315998e8-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.182519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.182507    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lbn5\" (UniqueName: \"kubernetes.io/projected/4c0648a8-63bf-4a92-8017-6921e4d12346-kube-api-access-5lbn5\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:30:15.537416 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.537321    2546 generic.go:356] "Generic (PLEG): container finished" podID="22d1388b-807e-4cbe-836f-b594315998e8" containerID="0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7" exitCode=0
Jul 20 10:30:15.537416 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.537395    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"
Jul 20 10:30:15.537416 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.537403    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerDied","Data":"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"}
Jul 20 10:30:15.538004 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.537464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn" event={"ID":"22d1388b-807e-4cbe-836f-b594315998e8","Type":"ContainerDied","Data":"89bec5524a65a1d75845ed2a7b335ae06c46005048de6c531adf9eb38c84c33a"}
Jul 20 10:30:15.538004 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.537492    2546 scope.go:117] "RemoveContainer" containerID="b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"
Jul 20 10:30:15.538982 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.538961    2546 generic.go:356] "Generic (PLEG): container finished" podID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerID="57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda" exitCode=0
Jul 20 10:30:15.539101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.538984    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerDied","Data":"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"}
Jul 20 10:30:15.539101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.539033    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv" event={"ID":"4c0648a8-63bf-4a92-8017-6921e4d12346","Type":"ContainerDied","Data":"15e30ca8caac04603f15a5a483d1f27d0b7ec7dd9ec706e310c5e892690a0d78"}
Jul 20 10:30:15.539101 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.539050    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"
Jul 20 10:30:15.548669 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.546545    2546 scope.go:117] "RemoveContainer" containerID="0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"
Jul 20 10:30:15.555882 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.555863    2546 scope.go:117] "RemoveContainer" containerID="b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"
Jul 20 10:30:15.556152 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:30:15.556125    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee\": container with ID starting with b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee not found: ID does not exist" containerID="b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"
Jul 20 10:30:15.556263 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.556160    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee"} err="failed to get container status \"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee\": rpc error: code = NotFound desc = could not find container \"b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee\": container with ID starting with b1fa640a5144a825d2dd4d08f2f1d6c332e52464b33bc4e0491760ff12b884ee not found: ID does not exist"
Jul 20 10:30:15.556263 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.556184    2546 scope.go:117] "RemoveContainer" containerID="0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"
Jul 20 10:30:15.556500 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:30:15.556482    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7\": container with ID starting with 0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7 not found: ID does not exist" containerID="0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"
Jul 20 10:30:15.556559 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.556507    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7"} err="failed to get container status \"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7\": rpc error: code = NotFound desc = could not find container \"0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7\": container with ID starting with 0b368d3f4d3051cc6f3f6f17b61eadba967eed1c8ac9fa4c36cddc713cd841e7 not found: ID does not exist"
Jul 20 10:30:15.556559 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.556523    2546 scope.go:117] "RemoveContainer" containerID="4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"
Jul 20 10:30:15.565603 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.565581    2546 scope.go:117] "RemoveContainer" containerID="57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"
Jul 20 10:30:15.568600 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.568580    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:30:15.570251 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.570228    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-53302-predictor-687d4d7db9-w9qtn"]
Jul 20 10:30:15.573681 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.573663    2546 scope.go:117] "RemoveContainer" containerID="4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"
Jul 20 10:30:15.573922 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:30:15.573899    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714\": container with ID starting with 4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714 not found: ID does not exist" containerID="4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"
Jul 20 10:30:15.574011 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.573926    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714"} err="failed to get container status \"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714\": rpc error: code = NotFound desc = could not find container \"4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714\": container with ID starting with 4c80ebedef0adc000d2ebaa58c91f232569204c92445f8cca5f0510dd16aa714 not found: ID does not exist"
Jul 20 10:30:15.574011 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.573944    2546 scope.go:117] "RemoveContainer" containerID="57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"
Jul 20 10:30:15.574220 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:30:15.574201    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda\": container with ID starting with 57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda not found: ID does not exist" containerID="57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"
Jul 20 10:30:15.574301 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.574223    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda"} err="failed to get container status \"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda\": rpc error: code = NotFound desc = could not find container \"57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda\": container with ID starting with 57a8b364d386c911fee6155a7e84ff96d9151ecc6077c7f402029dbf25deebda not found: ID does not exist"
Jul 20 10:30:15.581455 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.581431    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:30:15.583209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:15.583192    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53302-predictor-5b4fdf4964-w5cwv"]
Jul 20 10:30:16.404791 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:16.404755    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d1388b-807e-4cbe-836f-b594315998e8" path="/var/lib/kubelet/pods/22d1388b-807e-4cbe-836f-b594315998e8/volumes"
Jul 20 10:30:16.405221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:16.405206    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" path="/var/lib/kubelet/pods/4c0648a8-63bf-4a92-8017-6921e4d12346/volumes"
Jul 20 10:30:18.530136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:18.530108    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:18.530547 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:18.530521    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:19.535879 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:19.535853    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:30:19.536333 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:19.536284    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:28.531371 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:28.531331    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:29.536866 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:29.536824    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:38.530971 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:38.530926    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:39.537033 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:39.536994    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:48.530606 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:48.530566    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 20 10:30:49.537053 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:49.537017    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 20 10:30:58.531669 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:58.531632    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:30:59.537656 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:30:59.537626    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:31:11.169586 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.169555    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.224696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.224663    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:31:11.225088 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.225034    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" containerID="cri-o://807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57" gracePeriod=30
Jul 20 10:31:11.225240 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.225139    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kube-rbac-proxy" containerID="cri-o://91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9" gracePeriod=30
Jul 20 10:31:11.258635 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.258608    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.269963 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.269938    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:31:11.270242 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.270218    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container" containerID="cri-o://4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8" gracePeriod=30
Jul 20 10:31:11.270322 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.270293    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kube-rbac-proxy" containerID="cri-o://060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee" gracePeriod=30
Jul 20 10:31:11.327028 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.326999    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.332442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332417    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:31:11.332855 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332841    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container"
Jul 20 10:31:11.332902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332857    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container"
Jul 20 10:31:11.332902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332871    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container"
Jul 20 10:31:11.332902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332877    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container"
Jul 20 10:31:11.332902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332887    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kube-rbac-proxy"
Jul 20 10:31:11.332902 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332893    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kube-rbac-proxy"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332907    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kube-rbac-proxy"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332913    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kube-rbac-proxy"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332974    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kserve-container"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332982    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0648a8-63bf-4a92-8017-6921e4d12346" containerName="kube-rbac-proxy"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332989    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kube-rbac-proxy"
Jul 20 10:31:11.333054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.332995    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d1388b-807e-4cbe-836f-b594315998e8" containerName="kserve-container"
Jul 20 10:31:11.336405 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.336387    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.338972 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.338947    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78ab0-predictor-serving-cert\""
Jul 20 10:31:11.339284 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.339262    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78ab0-kube-rbac-proxy-sar-config\""
Jul 20 10:31:11.346052 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.346031    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:31:11.473255 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.473183    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:31:11.475448 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.475424    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhhn\" (UniqueName: \"kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.475570 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.475460    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.475570 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.475513    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.476683 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.476668    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.480046 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.480028    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78ab0-kube-rbac-proxy-sar-config\""
Jul 20 10:31:11.480138 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.480106    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78ab0-predictor-serving-cert\""
Jul 20 10:31:11.494278 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.494258    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:31:11.495035 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.495011    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.576270 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576238    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.576404 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576303    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.576404 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576342    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6l8t\" (UniqueName: \"kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.576404 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576374    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.576514 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576421    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhhn\" (UniqueName: \"kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.576514 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.576458    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.577116 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.577095    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.578777 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.578756    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.585928 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.585902    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhhn\" (UniqueName: \"kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn\") pod \"success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") " pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.590941 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.590918    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.647441 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.647414    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:11.677201 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.677176    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.677335 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:11.677317    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-serving-cert: secret "error-404-isvc-78ab0-predictor-serving-cert" not found
Jul 20 10:31:11.677402 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:11.677373    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls podName:45de7aec-00a2-4dd7-b579-9c7b6b655c67 nodeName:}" failed. No retries permitted until 2026-07-20 10:31:12.177358419 +0000 UTC m=+2056.224783141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls") pod "error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" (UID: "45de7aec-00a2-4dd7-b579-9c7b6b655c67") : secret "error-404-isvc-78ab0-predictor-serving-cert" not found
Jul 20 10:31:11.677402 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.677321    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.677530 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.677425    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6l8t\" (UniqueName: \"kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.677891 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.677870    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.685709 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.685684    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6l8t\" (UniqueName: \"kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:11.787082 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.787035    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:11.808675 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.808649    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:31:11.811025 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:31:11.810996    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d16fee0_3b7f_42a4_a8a1_52867747c7c0.slice/crio-5840e604529c76195dd51deed87704f4fdd39f708cd774ee2bb5ff468397259b WatchSource:0}: Error finding container 5840e604529c76195dd51deed87704f4fdd39f708cd774ee2bb5ff468397259b: Status 404 returned error can't find the container with id 5840e604529c76195dd51deed87704f4fdd39f708cd774ee2bb5ff468397259b
Jul 20 10:31:11.813903 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.813879    2546 generic.go:356] "Generic (PLEG): container finished" podID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerID="060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee" exitCode=2
Jul 20 10:31:11.814010 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.813956    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerDied","Data":"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"}
Jul 20 10:31:11.815784 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.815765    2546 generic.go:356] "Generic (PLEG): container finished" podID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerID="91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9" exitCode=2
Jul 20 10:31:11.815870 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.815805    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerDied","Data":"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"}
Jul 20 10:31:11.858142 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:11.858120    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:12.096573 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.096545    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:12.181772 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.181740    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:12.184122 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.184103    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") pod \"error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") " pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:12.387209 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.387129    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:12.520529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.520504    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:31:12.522567 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:31:12.522539    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45de7aec_00a2_4dd7_b579_9c7b6b655c67.slice/crio-e63c1160ef03492d97954a80b847f92386288972865fb4f6ee47bb71f22453df WatchSource:0}: Error finding container e63c1160ef03492d97954a80b847f92386288972865fb4f6ee47bb71f22453df: Status 404 returned error can't find the container with id e63c1160ef03492d97954a80b847f92386288972865fb4f6ee47bb71f22453df
Jul 20 10:31:12.823366 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.823332    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerStarted","Data":"6e8e079caadd051e9136562416dad2f90534ad6c55597cd7c95e84bc6ca3307d"}
Jul 20 10:31:12.823536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.823387    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerStarted","Data":"b73aa8b408398ccbf51bcc4ec7833210fda2e0fce36add6835fe6e735714e4cc"}
Jul 20 10:31:12.823536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.823421    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerStarted","Data":"e63c1160ef03492d97954a80b847f92386288972865fb4f6ee47bb71f22453df"}
Jul 20 10:31:12.823536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.823525    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:12.823715 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.823567    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:12.825186 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825156    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:12.825372 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825344    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerStarted","Data":"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"}
Jul 20 10:31:12.825437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825389    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:12.825437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825410    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerStarted","Data":"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"}
Jul 20 10:31:12.825437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825430    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:12.825549 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.825449    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerStarted","Data":"5840e604529c76195dd51deed87704f4fdd39f708cd774ee2bb5ff468397259b"}
Jul 20 10:31:12.826504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.826480    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:12.841178 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.841135    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podStartSLOduration=1.841123777 podStartE2EDuration="1.841123777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:12.839621341 +0000 UTC m=+2056.887046088" watchObservedRunningTime="2026-07-20 10:31:12.841123777 +0000 UTC m=+2056.888548522"
Jul 20 10:31:12.858017 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:12.857968    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podStartSLOduration=1.857952542 podStartE2EDuration="1.857952542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:12.856999755 +0000 UTC m=+2056.904424524" watchObservedRunningTime="2026-07-20 10:31:12.857952542 +0000 UTC m=+2056.905377288"
Jul 20 10:31:13.831134 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:13.831089    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:13.831588 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:13.831163    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:14.145910 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.145820    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 20 10:31:14.157443 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.157410    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 20 10:31:14.677926 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.677903    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:31:14.806402 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.806327    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls\") pod \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") "
Jul 20 10:31:14.806402 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.806361    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config\") pod \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") "
Jul 20 10:31:14.806402 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.806397    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvb2j\" (UniqueName: \"kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j\") pod \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\" (UID: \"ecfbf234-83ae-48f9-bfbe-cefac206efc7\") "
Jul 20 10:31:14.806766 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.806732    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecfbf234-83ae-48f9-bfbe-cefac206efc7" (UID: "ecfbf234-83ae-48f9-bfbe-cefac206efc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:31:14.808454 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.808429    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecfbf234-83ae-48f9-bfbe-cefac206efc7" (UID: "ecfbf234-83ae-48f9-bfbe-cefac206efc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:31:14.808535 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.808482    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j" (OuterVolumeSpecName: "kube-api-access-xvb2j") pod "ecfbf234-83ae-48f9-bfbe-cefac206efc7" (UID: "ecfbf234-83ae-48f9-bfbe-cefac206efc7"). InnerVolumeSpecName "kube-api-access-xvb2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:31:14.837675 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.837645    2546 generic.go:356] "Generic (PLEG): container finished" podID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerID="807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57" exitCode=0
Jul 20 10:31:14.838043 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.837703    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"
Jul 20 10:31:14.838043 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.837728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerDied","Data":"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"}
Jul 20 10:31:14.838043 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.837811    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm" event={"ID":"ecfbf234-83ae-48f9-bfbe-cefac206efc7","Type":"ContainerDied","Data":"8a9bc668d09d94e54f2e1e8ba3fb90729757db1d2011942d27d81bd1e2a4f972"}
Jul 20 10:31:14.838043 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.837850    2546 scope.go:117] "RemoveContainer" containerID="91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"
Jul 20 10:31:14.849531 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.849511    2546 scope.go:117] "RemoveContainer" containerID="807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"
Jul 20 10:31:14.857752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.857705    2546 scope.go:117] "RemoveContainer" containerID="91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"
Jul 20 10:31:14.858036 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:14.857987    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9\": container with ID starting with 91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9 not found: ID does not exist" containerID="91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"
Jul 20 10:31:14.858131 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.858033    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9"} err="failed to get container status \"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9\": rpc error: code = NotFound desc = could not find container \"91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9\": container with ID starting with 91e0b61ff91cf43320ed29a8ec4cb1a1cb1df9fb7c1e2b4456a10e15eb3537b9 not found: ID does not exist"
Jul 20 10:31:14.858131 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.858060    2546 scope.go:117] "RemoveContainer" containerID="807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"
Jul 20 10:31:14.858373 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:14.858349    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57\": container with ID starting with 807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57 not found: ID does not exist" containerID="807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"
Jul 20 10:31:14.858451 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.858394    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57"} err="failed to get container status \"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57\": rpc error: code = NotFound desc = could not find container \"807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57\": container with ID starting with 807affd228a120188564a4b3b14744f2900136c7087a1d89df928ac433a07f57 not found: ID does not exist"
Jul 20 10:31:14.862834 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.862813    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:31:14.866333 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.866310    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-64e78-predictor-b57c5f466-7vqfm"]
Jul 20 10:31:14.908292 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.908263    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecfbf234-83ae-48f9-bfbe-cefac206efc7-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:14.908292 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.908295    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:14.908465 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:14.908313    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvb2j\" (UniqueName: \"kubernetes.io/projected/ecfbf234-83ae-48f9-bfbe-cefac206efc7-kube-api-access-xvb2j\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:15.027890 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.027864    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:31:15.114523 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.114495    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") pod \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") "
Jul 20 10:31:15.114692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.114537    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g95tk\" (UniqueName: \"kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk\") pod \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") "
Jul 20 10:31:15.114692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.114574    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config\") pod \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\" (UID: \"50e96327-c06e-4abc-b3b7-4c3992a3a34f\") "
Jul 20 10:31:15.114979 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.114955    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50e96327-c06e-4abc-b3b7-4c3992a3a34f" (UID: "50e96327-c06e-4abc-b3b7-4c3992a3a34f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:31:15.116514 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.116494    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk" (OuterVolumeSpecName: "kube-api-access-g95tk") pod "50e96327-c06e-4abc-b3b7-4c3992a3a34f" (UID: "50e96327-c06e-4abc-b3b7-4c3992a3a34f"). InnerVolumeSpecName "kube-api-access-g95tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:31:15.116585 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.116515    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50e96327-c06e-4abc-b3b7-4c3992a3a34f" (UID: "50e96327-c06e-4abc-b3b7-4c3992a3a34f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:31:15.215708 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.215681    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95tk\" (UniqueName: \"kubernetes.io/projected/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-api-access-g95tk\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:15.215708 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.215702    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50e96327-c06e-4abc-b3b7-4c3992a3a34f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:15.215708 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.215712    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50e96327-c06e-4abc-b3b7-4c3992a3a34f-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:15.844581 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.844549    2546 generic.go:356] "Generic (PLEG): container finished" podID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerID="4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8" exitCode=0
Jul 20 10:31:15.845054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.844603    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"
Jul 20 10:31:15.845054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.844627    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerDied","Data":"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"}
Jul 20 10:31:15.845054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.844684    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql" event={"ID":"50e96327-c06e-4abc-b3b7-4c3992a3a34f","Type":"ContainerDied","Data":"93f3e89d8a457aed0e895936bf17b3187e1f4a73abdb0c2aae4a57ae72664807"}
Jul 20 10:31:15.845054 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.844713    2546 scope.go:117] "RemoveContainer" containerID="060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"
Jul 20 10:31:15.860358 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.860330    2546 scope.go:117] "RemoveContainer" containerID="4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"
Jul 20 10:31:15.868671 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.868649    2546 scope.go:117] "RemoveContainer" containerID="060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"
Jul 20 10:31:15.868894 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:15.868874    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee\": container with ID starting with 060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee not found: ID does not exist" containerID="060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"
Jul 20 10:31:15.869004 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.868898    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee"} err="failed to get container status \"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee\": rpc error: code = NotFound desc = could not find container \"060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee\": container with ID starting with 060ef21e3dc55293e778c6239c5584d99b04606d578344bfa10d6081193439ee not found: ID does not exist"
Jul 20 10:31:15.869004 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.868919    2546 scope.go:117] "RemoveContainer" containerID="4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"
Jul 20 10:31:15.869239 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:15.869210    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8\": container with ID starting with 4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8 not found: ID does not exist" containerID="4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"
Jul 20 10:31:15.869314 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.869244    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8"} err="failed to get container status \"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8\": rpc error: code = NotFound desc = could not find container \"4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8\": container with ID starting with 4c2deb08d7c3e6b4bc380ac842881e5a21d67aebc76bdc00f7531d54aa3e4eb8 not found: ID does not exist"
Jul 20 10:31:15.871854 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.871830    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:31:15.875855 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:15.875831    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-64e78-predictor-676cf96d7f-fsxql"]
Jul 20 10:31:16.404044 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:16.404013    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" path="/var/lib/kubelet/pods/50e96327-c06e-4abc-b3b7-4c3992a3a34f/volumes"
Jul 20 10:31:16.404462 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:16.404446    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" path="/var/lib/kubelet/pods/ecfbf234-83ae-48f9-bfbe-cefac206efc7/volumes"
Jul 20 10:31:18.835312 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:18.835285    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:31:18.835749 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:18.835645    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:18.835808 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:18.835760    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:18.836057 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:18.836032    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:21.903088 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:21.903030    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:21.989823 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:21.989791    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:31:21.990133 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:21.990094    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container" containerID="cri-o://7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2" gracePeriod=30
Jul 20 10:31:21.990133 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:21.990114    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kube-rbac-proxy" containerID="cri-o://38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198" gracePeriod=30
Jul 20 10:31:22.037987 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.037938    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:31:22.038334 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.038297    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container" containerID="cri-o://4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2" gracePeriod=30
Jul 20 10:31:22.038825 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.038670    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kube-rbac-proxy" containerID="cri-o://d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798" gracePeriod=30
Jul 20 10:31:22.069189 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069163    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:31:22.069576 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069564    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069578    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069608    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container"
Jul 20 10:31:22.069638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069613    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container"
Jul 20 10:31:22.069638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069620    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069626    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069641    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069647    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069698    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069706    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kserve-container"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069716    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecfbf234-83ae-48f9-bfbe-cefac206efc7" containerName="kube-rbac-proxy"
Jul 20 10:31:22.069796 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.069723    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="50e96327-c06e-4abc-b3b7-4c3992a3a34f" containerName="kserve-container"
Jul 20 10:31:22.074398 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.074380    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.077172 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.077148    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6e423-predictor-serving-cert\""
Jul 20 10:31:22.077288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.077157    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6e423-kube-rbac-proxy-sar-config\""
Jul 20 10:31:22.083825 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.083801    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:31:22.168821 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.168793    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:31:22.172565 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.172546    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.175185 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.175166    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6e423-predictor-serving-cert\""
Jul 20 10:31:22.175297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.175167    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6e423-kube-rbac-proxy-sar-config\""
Jul 20 10:31:22.175467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.175447    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.175529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.175482    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.175582 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.175556    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlhx\" (UniqueName: \"kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.182221 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.182200    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:31:22.276515 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276485    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjx8b\" (UniqueName: \"kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.276678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276519    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.276678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276601    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.276678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276620    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.276678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276638    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.276873 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.276694    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlhx\" (UniqueName: \"kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.276873 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:22.276841    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6e423-predictor-serving-cert: secret "success-200-isvc-6e423-predictor-serving-cert" not found
Jul 20 10:31:22.276980 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:22.276917    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls podName:2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f nodeName:}" failed. No retries permitted until 2026-07-20 10:31:22.776895175 +0000 UTC m=+2066.824319897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls") pod "success-200-isvc-6e423-predictor-6b546f494c-972tv" (UID: "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f") : secret "success-200-isvc-6e423-predictor-serving-cert" not found
Jul 20 10:31:22.277301 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.277283    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.286189 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.286166    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlhx\" (UniqueName: \"kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.378103 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.378053    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjx8b\" (UniqueName: \"kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.378268 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.378107    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.378268 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.378185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.378360 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:22.378303    2546 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-6e423-predictor-serving-cert: secret "error-404-isvc-6e423-predictor-serving-cert" not found
Jul 20 10:31:22.378403 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:22.378373    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls podName:9b0f6d24-13a4-475c-88de-e8b0a6e9767b nodeName:}" failed. No retries permitted until 2026-07-20 10:31:22.878359058 +0000 UTC m=+2066.925783780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls") pod "error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" (UID: "9b0f6d24-13a4-475c-88de-e8b0a6e9767b") : secret "error-404-isvc-6e423-predictor-serving-cert" not found
Jul 20 10:31:22.378782 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.378764    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.388471 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.388451    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjx8b\" (UniqueName: \"kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.781421 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.781378    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.783817 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.783791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") pod \"success-200-isvc-6e423-predictor-6b546f494c-972tv\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") " pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:22.881838 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.881809    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.883531 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.883506    2546 generic.go:356] "Generic (PLEG): container finished" podID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerID="d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798" exitCode=2
Jul 20 10:31:22.883672 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.883587    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerDied","Data":"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"}
Jul 20 10:31:22.884392 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.884369    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") pod \"error-404-isvc-6e423-predictor-5dbc84fc58-rktlq\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") " pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:22.885450 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.885431    2546 generic.go:356] "Generic (PLEG): container finished" podID="002f075b-ee91-4ce2-b135-2704a2312877" containerID="38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198" exitCode=2
Jul 20 10:31:22.885561 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.885503    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerDied","Data":"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"}
Jul 20 10:31:22.985522 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:22.985491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:23.084011 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.083977    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:23.121742 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.121535    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:31:23.122977 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:31:23.122951    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5b2dfd_f4b6_447f_b917_b8b38d2d6d3f.slice/crio-821a7bc2bf51f373abc7dbfab41c78f76f62fe0f9e5513ee537ef0da634469bf WatchSource:0}: Error finding container 821a7bc2bf51f373abc7dbfab41c78f76f62fe0f9e5513ee537ef0da634469bf: Status 404 returned error can't find the container with id 821a7bc2bf51f373abc7dbfab41c78f76f62fe0f9e5513ee537ef0da634469bf
Jul 20 10:31:23.229437 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.229400    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:31:23.241132 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:31:23.241103    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0f6d24_13a4_475c_88de_e8b0a6e9767b.slice/crio-2e8d454c2df0560f3302ea37cb58d29cb1c9b4a75d830a58de016f56df6625c1 WatchSource:0}: Error finding container 2e8d454c2df0560f3302ea37cb58d29cb1c9b4a75d830a58de016f56df6625c1: Status 404 returned error can't find the container with id 2e8d454c2df0560f3302ea37cb58d29cb1c9b4a75d830a58de016f56df6625c1
Jul 20 10:31:23.526317 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.526279    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 20 10:31:23.739219 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.739187    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:31:23.892597 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.892555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerStarted","Data":"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"}
Jul 20 10:31:23.892761 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.892622    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerStarted","Data":"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"}
Jul 20 10:31:23.892822 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.892806    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:23.892882 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.892861    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerStarted","Data":"821a7bc2bf51f373abc7dbfab41c78f76f62fe0f9e5513ee537ef0da634469bf"}
Jul 20 10:31:23.892955 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.892903    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:23.894419 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894391    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:31:23.894664 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894646    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerStarted","Data":"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"}
Jul 20 10:31:23.894714 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894692    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerStarted","Data":"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"}
Jul 20 10:31:23.894747 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894717    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:23.894747 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894736    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerStarted","Data":"2e8d454c2df0560f3302ea37cb58d29cb1c9b4a75d830a58de016f56df6625c1"}
Jul 20 10:31:23.894848 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.894763    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:23.895698 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.895677    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:23.913346 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.913308    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podStartSLOduration=1.9132945609999998 podStartE2EDuration="1.913294561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:23.911247807 +0000 UTC m=+2067.958672551" watchObservedRunningTime="2026-07-20 10:31:23.913294561 +0000 UTC m=+2067.960719306"
Jul 20 10:31:23.929650 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:23.929602    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podStartSLOduration=1.929589191 podStartE2EDuration="1.929589191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:31:23.928112258 +0000 UTC m=+2067.975537004" watchObservedRunningTime="2026-07-20 10:31:23.929589191 +0000 UTC m=+2067.977013935"
Jul 20 10:31:24.531645 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:24.531606    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 20 10:31:24.899608 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:24.899565    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:31:24.899811 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:24.899649    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:25.700831 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.700808    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:31:25.750638 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.750614    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:31:25.812116 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.812084    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvbxl\" (UniqueName: \"kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl\") pod \"53ee06bf-5653-4a01-9147-c0adf85c66a5\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") "
Jul 20 10:31:25.812262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.812231    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config\") pod \"53ee06bf-5653-4a01-9147-c0adf85c66a5\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") "
Jul 20 10:31:25.812320 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.812282    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls\") pod \"53ee06bf-5653-4a01-9147-c0adf85c66a5\" (UID: \"53ee06bf-5653-4a01-9147-c0adf85c66a5\") "
Jul 20 10:31:25.812628 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.812598    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53ee06bf-5653-4a01-9147-c0adf85c66a5" (UID: "53ee06bf-5653-4a01-9147-c0adf85c66a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:31:25.814017 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.813999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl" (OuterVolumeSpecName: "kube-api-access-vvbxl") pod "53ee06bf-5653-4a01-9147-c0adf85c66a5" (UID: "53ee06bf-5653-4a01-9147-c0adf85c66a5"). InnerVolumeSpecName "kube-api-access-vvbxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:31:25.814187 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.814167    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53ee06bf-5653-4a01-9147-c0adf85c66a5" (UID: "53ee06bf-5653-4a01-9147-c0adf85c66a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:31:25.906696 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.906668    2546 generic.go:356] "Generic (PLEG): container finished" podID="002f075b-ee91-4ce2-b135-2704a2312877" containerID="7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2" exitCode=0
Jul 20 10:31:25.906819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.906715    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"
Jul 20 10:31:25.906819 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.906746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerDied","Data":"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"}
Jul 20 10:31:25.906905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.906829    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552" event={"ID":"002f075b-ee91-4ce2-b135-2704a2312877","Type":"ContainerDied","Data":"d9c30b21b60045e1985c69a72be10cfaace85845195fc76380d52ff4511d81cc"}
Jul 20 10:31:25.906905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.906855    2546 scope.go:117] "RemoveContainer" containerID="38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"
Jul 20 10:31:25.908321 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.908295    2546 generic.go:356] "Generic (PLEG): container finished" podID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerID="4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2" exitCode=0
Jul 20 10:31:25.908401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.908331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerDied","Data":"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"}
Jul 20 10:31:25.908401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.908350    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"
Jul 20 10:31:25.908401 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.908377    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj" event={"ID":"53ee06bf-5653-4a01-9147-c0adf85c66a5","Type":"ContainerDied","Data":"60e13478c7fca9884cdc40dafff963e3c7758b539bbb97dfa79d5bbaf3ebf98d"}
Jul 20 10:31:25.912820 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.912800    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls\") pod \"002f075b-ee91-4ce2-b135-2704a2312877\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") "
Jul 20 10:31:25.912923 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.912864    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config\") pod \"002f075b-ee91-4ce2-b135-2704a2312877\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") "
Jul 20 10:31:25.913040 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913022    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdppz\" (UniqueName: \"kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz\") pod \"002f075b-ee91-4ce2-b135-2704a2312877\" (UID: \"002f075b-ee91-4ce2-b135-2704a2312877\") "
Jul 20 10:31:25.913215 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913191    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "002f075b-ee91-4ce2-b135-2704a2312877" (UID: "002f075b-ee91-4ce2-b135-2704a2312877"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:31:25.913349 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913330    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:25.913442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913352    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53ee06bf-5653-4a01-9147-c0adf85c66a5-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:25.913442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913364    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/002f075b-ee91-4ce2-b135-2704a2312877-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:25.913442 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.913374    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbxl\" (UniqueName: \"kubernetes.io/projected/53ee06bf-5653-4a01-9147-c0adf85c66a5-kube-api-access-vvbxl\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:25.915150 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.915129    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "002f075b-ee91-4ce2-b135-2704a2312877" (UID: "002f075b-ee91-4ce2-b135-2704a2312877"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:31:25.915257 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.915239    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz" (OuterVolumeSpecName: "kube-api-access-fdppz") pod "002f075b-ee91-4ce2-b135-2704a2312877" (UID: "002f075b-ee91-4ce2-b135-2704a2312877"). InnerVolumeSpecName "kube-api-access-fdppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:31:25.916698 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.916651    2546 scope.go:117] "RemoveContainer" containerID="7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"
Jul 20 10:31:25.924061 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.924046    2546 scope.go:117] "RemoveContainer" containerID="38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"
Jul 20 10:31:25.924333 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:25.924315    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198\": container with ID starting with 38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198 not found: ID does not exist" containerID="38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"
Jul 20 10:31:25.924397 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.924340    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198"} err="failed to get container status \"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198\": rpc error: code = NotFound desc = could not find container \"38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198\": container with ID starting with 38a89da34cae7994bc9b8fe840dfe686d68e4bceff044447c89eaaa5aebf4198 not found: ID does not exist"
Jul 20 10:31:25.924397 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.924354    2546 scope.go:117] "RemoveContainer" containerID="7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"
Jul 20 10:31:25.924583 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:25.924554    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2\": container with ID starting with 7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2 not found: ID does not exist" containerID="7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"
Jul 20 10:31:25.924662 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.924583    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2"} err="failed to get container status \"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2\": rpc error: code = NotFound desc = could not find container \"7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2\": container with ID starting with 7d9e29efd6a0cd791b9b418159f27672c6505a14c72af68d90460448f05e83d2 not found: ID does not exist"
Jul 20 10:31:25.924662 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.924599    2546 scope.go:117] "RemoveContainer" containerID="d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"
Jul 20 10:31:25.932623 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.932607    2546 scope.go:117] "RemoveContainer" containerID="4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"
Jul 20 10:31:25.933338 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.933317    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:31:25.939922 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.939903    2546 scope.go:117] "RemoveContainer" containerID="d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"
Jul 20 10:31:25.940185 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:25.940167    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798\": container with ID starting with d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798 not found: ID does not exist" containerID="d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"
Jul 20 10:31:25.940267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.940193    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798"} err="failed to get container status \"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798\": rpc error: code = NotFound desc = could not find container \"d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798\": container with ID starting with d6277f3856afc8dcfe5c0770f3f60becb4904256301275171ffe69f49a884798 not found: ID does not exist"
Jul 20 10:31:25.940267 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.940213    2546 scope.go:117] "RemoveContainer" containerID="4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"
Jul 20 10:31:25.940460 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:31:25.940443    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2\": container with ID starting with 4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2 not found: ID does not exist" containerID="4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"
Jul 20 10:31:25.940504 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.940465    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2"} err="failed to get container status \"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2\": rpc error: code = NotFound desc = could not find container \"4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2\": container with ID starting with 4eaad9e3dabe31b2b1b1d5dbfd285ebdfa82274432841cb92f08118a679baec2 not found: ID does not exist"
Jul 20 10:31:25.940954 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:25.940935    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a956e-predictor-78896946fd-x7gsj"]
Jul 20 10:31:26.014093 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.013999    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdppz\" (UniqueName: \"kubernetes.io/projected/002f075b-ee91-4ce2-b135-2704a2312877-kube-api-access-fdppz\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:26.014093 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.014025    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/002f075b-ee91-4ce2-b135-2704a2312877-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:31:26.229035 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.229009    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:31:26.233350 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.233326    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a956e-predictor-6f7cf8c9-8n552"]
Jul 20 10:31:26.404614 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.404584    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="002f075b-ee91-4ce2-b135-2704a2312877" path="/var/lib/kubelet/pods/002f075b-ee91-4ce2-b135-2704a2312877/volumes"
Jul 20 10:31:26.405018 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:26.405003    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" path="/var/lib/kubelet/pods/53ee06bf-5653-4a01-9147-c0adf85c66a5/volumes"
Jul 20 10:31:28.835848 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:28.835805    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:28.836256 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:28.835995    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:29.904792 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:29.904759    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:31:29.905297 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:29.905263    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:31:29.905575 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:29.905554    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:31:29.905943 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:29.905920    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:38.836412 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:38.836372    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:38.836864 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:38.836372    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:39.905281 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:39.905237    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:31:39.906440 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:39.906415    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:48.836523 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:48.836483    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:48.836523 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:48.836510    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 20 10:31:49.905560 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:49.905519    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:31:49.906002 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:49.905960    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:56.560014 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:56.559979    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:31:56.561518 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:56.561493    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:31:56.563862 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:56.563840    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:31:56.565417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:56.565399    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:31:58.835906 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:58.835862    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 20 10:31:58.836429 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:58.836201    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:31:59.906275 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:59.906238    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:31:59.906668 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:31:59.906238    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:32:08.836663 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:08.836635    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:32:09.905576 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:09.905527    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:32:09.906207 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:09.906188    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:32:11.555594 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:11.555556    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:32:11.905277 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:11.905242    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:32:19.906113 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:19.906081    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:32:32.341968 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:32.341933    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:32:32.431390 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:32:32.431354    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:33:05.340766 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:33:05.340690    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:35:53.027618 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:35:53.027518    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:36:04.130570 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:04.130535    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:36:15.680454 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:15.680419    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:36:15.774637 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:15.774602    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:36:56.595844 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:56.595813    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:36:56.597018 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:56.596994    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:36:56.600000 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:56.599978    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:36:56.601098 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:36:56.601058    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:40:46.825224 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.825179    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:46.898191 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.898156    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:40:46.898428 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.898400    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" containerID="cri-o://cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c" gracePeriod=30
Jul 20 10:40:46.898552 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.898428    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kube-rbac-proxy" containerID="cri-o://8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb" gracePeriod=30
Jul 20 10:40:46.940023 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.939989    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:40:46.940275 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.940233    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" containerID="cri-o://f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf" gracePeriod=30
Jul 20 10:40:46.940387 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:46.940265    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kube-rbac-proxy" containerID="cri-o://1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef" gracePeriod=30
Jul 20 10:40:47.610288 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:47.610250    2546 generic.go:356] "Generic (PLEG): container finished" podID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerID="8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb" exitCode=2
Jul 20 10:40:47.610491 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:47.610318    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerDied","Data":"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"}
Jul 20 10:40:47.612037 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:47.612015    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerID="1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef" exitCode=2
Jul 20 10:40:47.612126 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:47.612088    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerDied","Data":"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"}
Jul 20 10:40:47.833553 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:47.833520    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:40:49.900310 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:49.900276    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 20 10:40:49.900310 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:49.900291    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 20 10:40:49.905710 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:49.905686    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 20 10:40:49.905945 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:49.905921    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 20 10:40:50.079860 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.079833    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:40:50.110780 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.110718    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:40:50.194343 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194311    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qlhx\" (UniqueName: \"kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx\") pod \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") "
Jul 20 10:40:50.194508 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194349    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") pod \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") "
Jul 20 10:40:50.194508 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194391    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config\") pod \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") "
Jul 20 10:40:50.194508 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194422    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjx8b\" (UniqueName: \"kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b\") pod \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") "
Jul 20 10:40:50.194508 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194451    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config\") pod \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\" (UID: \"9b0f6d24-13a4-475c-88de-e8b0a6e9767b\") "
Jul 20 10:40:50.194678 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194638    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") pod \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\" (UID: \"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f\") "
Jul 20 10:40:50.194856 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194826    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b0f6d24-13a4-475c-88de-e8b0a6e9767b" (UID: "9b0f6d24-13a4-475c-88de-e8b0a6e9767b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:40:50.194856 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194833    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" (UID: "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:40:50.195046 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194959    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.195046 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.194980    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.196588 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.196566    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b0f6d24-13a4-475c-88de-e8b0a6e9767b" (UID: "9b0f6d24-13a4-475c-88de-e8b0a6e9767b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:50.196695 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.196668    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" (UID: "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:40:50.196997 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.196980    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b" (OuterVolumeSpecName: "kube-api-access-tjx8b") pod "9b0f6d24-13a4-475c-88de-e8b0a6e9767b" (UID: "9b0f6d24-13a4-475c-88de-e8b0a6e9767b"). InnerVolumeSpecName "kube-api-access-tjx8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:50.197042 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.197003    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx" (OuterVolumeSpecName: "kube-api-access-9qlhx") pod "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" (UID: "2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f"). InnerVolumeSpecName "kube-api-access-9qlhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:40:50.295529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.295500    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlhx\" (UniqueName: \"kubernetes.io/projected/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-kube-api-access-9qlhx\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.295529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.295524    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.295529 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.295534    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjx8b\" (UniqueName: \"kubernetes.io/projected/9b0f6d24-13a4-475c-88de-e8b0a6e9767b-kube-api-access-tjx8b\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.295751 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.295543    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:40:50.627925 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.627884    2546 generic.go:356] "Generic (PLEG): container finished" podID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerID="cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c" exitCode=0
Jul 20 10:40:50.628136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.627930    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"
Jul 20 10:40:50.628136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.627975    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerDied","Data":"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"}
Jul 20 10:40:50.628136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.628030    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv" event={"ID":"2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f","Type":"ContainerDied","Data":"821a7bc2bf51f373abc7dbfab41c78f76f62fe0f9e5513ee537ef0da634469bf"}
Jul 20 10:40:50.628136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.628088    2546 scope.go:117] "RemoveContainer" containerID="8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"
Jul 20 10:40:50.629636 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.629612    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerID="f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf" exitCode=0
Jul 20 10:40:50.629752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.629658    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"
Jul 20 10:40:50.629752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.629708    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerDied","Data":"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"}
Jul 20 10:40:50.629874 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.629758    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq" event={"ID":"9b0f6d24-13a4-475c-88de-e8b0a6e9767b","Type":"ContainerDied","Data":"2e8d454c2df0560f3302ea37cb58d29cb1c9b4a75d830a58de016f56df6625c1"}
Jul 20 10:40:50.636869 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.636851    2546 scope.go:117] "RemoveContainer" containerID="cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"
Jul 20 10:40:50.644791 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.644767    2546 scope.go:117] "RemoveContainer" containerID="8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"
Jul 20 10:40:50.645084 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:40:50.645048    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb\": container with ID starting with 8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb not found: ID does not exist" containerID="8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"
Jul 20 10:40:50.645161 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.645088    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb"} err="failed to get container status \"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb\": rpc error: code = NotFound desc = could not find container \"8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb\": container with ID starting with 8093dcc54070bf6c2ce9ba8cf8b0587bbc261638ae95a5ce3ecb2dc797e25adb not found: ID does not exist"
Jul 20 10:40:50.645161 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.645106    2546 scope.go:117] "RemoveContainer" containerID="cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"
Jul 20 10:40:50.645356 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:40:50.645336    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c\": container with ID starting with cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c not found: ID does not exist" containerID="cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"
Jul 20 10:40:50.645397 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.645364    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c"} err="failed to get container status \"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c\": rpc error: code = NotFound desc = could not find container \"cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c\": container with ID starting with cea2aa9b9e67273c03d40678ef019b9b47c355e213295d92dcb94cab60f6493c not found: ID does not exist"
Jul 20 10:40:50.645397 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.645380    2546 scope.go:117] "RemoveContainer" containerID="1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"
Jul 20 10:40:50.646905 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.646876    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:40:50.651464 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.651443    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6e423-predictor-6b546f494c-972tv"]
Jul 20 10:40:50.653537 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.653522    2546 scope.go:117] "RemoveContainer" containerID="f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"
Jul 20 10:40:50.661028 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.660987    2546 scope.go:117] "RemoveContainer" containerID="1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"
Jul 20 10:40:50.661378 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:40:50.661336    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef\": container with ID starting with 1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef not found: ID does not exist" containerID="1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"
Jul 20 10:40:50.661467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.661380    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef"} err="failed to get container status \"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef\": rpc error: code = NotFound desc = could not find container \"1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef\": container with ID starting with 1fc701a51f41cedcb6515404623b31655a164b1cbb0b107549b1e0f8e1dca9ef not found: ID does not exist"
Jul 20 10:40:50.661467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.661402    2546 scope.go:117] "RemoveContainer" containerID="f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"
Jul 20 10:40:50.661741 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:40:50.661708    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf\": container with ID starting with f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf not found: ID does not exist" containerID="f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"
Jul 20 10:40:50.661794 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.661737    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf"} err="failed to get container status \"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf\": rpc error: code = NotFound desc = could not find container \"f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf\": container with ID starting with f384b33ba22c3963da6b729a9dfd71cec10208d93154a95ac8a229413e8175bf not found: ID does not exist"
Jul 20 10:40:50.663119 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.663099    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:40:50.669620 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:50.669572    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6e423-predictor-5dbc84fc58-rktlq"]
Jul 20 10:40:52.403667 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:52.403623    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" path="/var/lib/kubelet/pods/2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f/volumes"
Jul 20 10:40:52.404270 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:40:52.404248    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" path="/var/lib/kubelet/pods/9b0f6d24-13a4-475c-88de-e8b0a6e9767b/volumes"
Jul 20 10:41:56.624566 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:41:56.624534    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:41:56.625574 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:41:56.625556    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:41:56.629734 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:41:56.629713    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:41:56.630811 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:41:56.630793    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:42:28.618443 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:42:28.618403    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:43:26.404125 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:43:26.404093    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:46:56.655519 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:46:56.655487    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:46:56.657997 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:46:56.656755    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:46:56.661182 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:46:56.661161    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:46:56.662480 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:46:56.662455    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:48:30.602038 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.602000    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:30.699790 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.699756    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:48:30.700018 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.699973    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container" containerID="cri-o://b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91" gracePeriod=30
Jul 20 10:48:30.700104 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.700026    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kube-rbac-proxy" containerID="cri-o://42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488" gracePeriod=30
Jul 20 10:48:30.757967 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.757933    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:48:30.758268 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.758240    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kube-rbac-proxy" containerID="cri-o://6e8e079caadd051e9136562416dad2f90534ad6c55597cd7c95e84bc6ca3307d" gracePeriod=30
Jul 20 10:48:30.759118 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:30.759082    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container" containerID="cri-o://b73aa8b408398ccbf51bcc4ec7833210fda2e0fce36add6835fe6e735714e4cc" gracePeriod=30
Jul 20 10:48:31.753646 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:31.753611    2546 generic.go:356] "Generic (PLEG): container finished" podID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerID="6e8e079caadd051e9136562416dad2f90534ad6c55597cd7c95e84bc6ca3307d" exitCode=2
Jul 20 10:48:31.754087 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:31.753682    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerDied","Data":"6e8e079caadd051e9136562416dad2f90534ad6c55597cd7c95e84bc6ca3307d"}
Jul 20 10:48:31.755348 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:31.755321    2546 generic.go:356] "Generic (PLEG): container finished" podID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerID="42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488" exitCode=2
Jul 20 10:48:31.755467 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:31.755389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerDied","Data":"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"}
Jul 20 10:48:31.950882 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:31.950842    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:32.191686 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:32.191649    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:32.293630 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:32.293603    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:48:33.753040 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.753016    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:48:33.768766 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.768730    2546 generic.go:356] "Generic (PLEG): container finished" podID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerID="b73aa8b408398ccbf51bcc4ec7833210fda2e0fce36add6835fe6e735714e4cc" exitCode=0
Jul 20 10:48:33.768928 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.768798    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerDied","Data":"b73aa8b408398ccbf51bcc4ec7833210fda2e0fce36add6835fe6e735714e4cc"}
Jul 20 10:48:33.770555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.770534    2546 generic.go:356] "Generic (PLEG): container finished" podID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerID="b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91" exitCode=0
Jul 20 10:48:33.770666 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.770604    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"
Jul 20 10:48:33.770666 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.770623    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerDied","Data":"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"}
Jul 20 10:48:33.770761 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.770679    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs" event={"ID":"9d16fee0-3b7f-42a4-a8a1-52867747c7c0","Type":"ContainerDied","Data":"5840e604529c76195dd51deed87704f4fdd39f708cd774ee2bb5ff468397259b"}
Jul 20 10:48:33.770761 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.770708    2546 scope.go:117] "RemoveContainer" containerID="42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"
Jul 20 10:48:33.781485 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.781463    2546 scope.go:117] "RemoveContainer" containerID="b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"
Jul 20 10:48:33.790521 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.790496    2546 scope.go:117] "RemoveContainer" containerID="42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"
Jul 20 10:48:33.790809 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:48:33.790789    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488\": container with ID starting with 42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488 not found: ID does not exist" containerID="42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"
Jul 20 10:48:33.790874 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.790817    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488"} err="failed to get container status \"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488\": rpc error: code = NotFound desc = could not find container \"42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488\": container with ID starting with 42e9c224d259f464f65db5975105bb4957b4b877811b390b83537aa608df8488 not found: ID does not exist"
Jul 20 10:48:33.790874 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.790835    2546 scope.go:117] "RemoveContainer" containerID="b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"
Jul 20 10:48:33.791107 ip-10-0-143-155 kubenswrapper[2546]: E0720 10:48:33.791053    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91\": container with ID starting with b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91 not found: ID does not exist" containerID="b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"
Jul 20 10:48:33.791107 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.791092    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91"} err="failed to get container status \"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91\": rpc error: code = NotFound desc = could not find container \"b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91\": container with ID starting with b84cd8164dea89d23f39ae45470349c76c6803e04c52db9bf6a6035b9df7af91 not found: ID does not exist"
Jul 20 10:48:33.818099 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.818027    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:48:33.858123 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.858095    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config\") pod \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") "
Jul 20 10:48:33.858262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.858191    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crhhn\" (UniqueName: \"kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn\") pod \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") "
Jul 20 10:48:33.858262 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.858242    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls\") pod \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\" (UID: \"9d16fee0-3b7f-42a4-a8a1-52867747c7c0\") "
Jul 20 10:48:33.858463 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.858443    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9d16fee0-3b7f-42a4-a8a1-52867747c7c0" (UID: "9d16fee0-3b7f-42a4-a8a1-52867747c7c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:48:33.858547 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.858533    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:33.860299 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.860277    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9d16fee0-3b7f-42a4-a8a1-52867747c7c0" (UID: "9d16fee0-3b7f-42a4-a8a1-52867747c7c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:48:33.860406 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.860387    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn" (OuterVolumeSpecName: "kube-api-access-crhhn") pod "9d16fee0-3b7f-42a4-a8a1-52867747c7c0" (UID: "9d16fee0-3b7f-42a4-a8a1-52867747c7c0"). InnerVolumeSpecName "kube-api-access-crhhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:48:33.959386 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959352    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6l8t\" (UniqueName: \"kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t\") pod \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") "
Jul 20 10:48:33.959542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959458    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config\") pod \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") "
Jul 20 10:48:33.959542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959496    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") pod \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\" (UID: \"45de7aec-00a2-4dd7-b579-9c7b6b655c67\") "
Jul 20 10:48:33.959734 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959717    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhhn\" (UniqueName: \"kubernetes.io/projected/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-kube-api-access-crhhn\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:33.959771 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959742    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9d16fee0-3b7f-42a4-a8a1-52867747c7c0-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:33.959771 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.959736    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45de7aec-00a2-4dd7-b579-9c7b6b655c67" (UID: "45de7aec-00a2-4dd7-b579-9c7b6b655c67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 10:48:33.961422 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.961403    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45de7aec-00a2-4dd7-b579-9c7b6b655c67" (UID: "45de7aec-00a2-4dd7-b579-9c7b6b655c67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 10:48:33.961542 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:33.961522    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t" (OuterVolumeSpecName: "kube-api-access-l6l8t") pod "45de7aec-00a2-4dd7-b579-9c7b6b655c67" (UID: "45de7aec-00a2-4dd7-b579-9c7b6b655c67"). InnerVolumeSpecName "kube-api-access-l6l8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 10:48:34.060476 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.060441    2546 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:34.060476 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.060468    2546 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45de7aec-00a2-4dd7-b579-9c7b6b655c67-proxy-tls\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:34.060646 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.060482    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6l8t\" (UniqueName: \"kubernetes.io/projected/45de7aec-00a2-4dd7-b579-9c7b6b655c67-kube-api-access-l6l8t\") on node \"ip-10-0-143-155.ec2.internal\" DevicePath \"\""
Jul 20 10:48:34.092080 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.092041    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:48:34.095660 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.095636    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ab0-predictor-77ccfb98c9-mgczs"]
Jul 20 10:48:34.403805 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.403776    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" path="/var/lib/kubelet/pods/9d16fee0-3b7f-42a4-a8a1-52867747c7c0/volumes"
Jul 20 10:48:34.776320 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.776234    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"
Jul 20 10:48:34.776320 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.776242    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9" event={"ID":"45de7aec-00a2-4dd7-b579-9c7b6b655c67","Type":"ContainerDied","Data":"e63c1160ef03492d97954a80b847f92386288972865fb4f6ee47bb71f22453df"}
Jul 20 10:48:34.776784 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.776376    2546 scope.go:117] "RemoveContainer" containerID="6e8e079caadd051e9136562416dad2f90534ad6c55597cd7c95e84bc6ca3307d"
Jul 20 10:48:34.784740 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.784718    2546 scope.go:117] "RemoveContainer" containerID="b73aa8b408398ccbf51bcc4ec7833210fda2e0fce36add6835fe6e735714e4cc"
Jul 20 10:48:34.793456 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.793427    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:48:34.795340 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:34.795319    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78ab0-predictor-5d9cbd79b6-p5wz9"]
Jul 20 10:48:36.408120 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:48:36.408090    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" path="/var/lib/kubelet/pods/45de7aec-00a2-4dd7-b579-9c7b6b655c67/volumes"
Jul 20 10:49:03.472325 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.472286    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:03.609809 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.609774    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f7trj/must-gather-xhtp4"]
Jul 20 10:49:03.610154 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610140    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container"
Jul 20 10:49:03.610203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610157    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container"
Jul 20 10:49:03.610203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610168    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container"
Jul 20 10:49:03.610203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610178    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container"
Jul 20 10:49:03.610203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610192    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container"
Jul 20 10:49:03.610203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610197    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610206    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610211    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610217    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610223    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610232    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610237    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610249    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610254    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610260    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610265    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610270    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610275    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610283    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610287    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610292    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610297    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610304    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610309    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610357 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610358    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610366    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610373    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="002f075b-ee91-4ce2-b135-2704a2312877" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610380    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610385    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ee06bf-5653-4a01-9147-c0adf85c66a5" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610390    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610396    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610400    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610405    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="45de7aec-00a2-4dd7-b579-9c7b6b655c67" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610409    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0f6d24-13a4-475c-88de-e8b0a6e9767b" containerName="kserve-container"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610415    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d16fee0-3b7f-42a4-a8a1-52867747c7c0" containerName="kube-rbac-proxy"
Jul 20 10:49:03.610921 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.610421    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5b2dfd-f4b6-447f-b917-b8b38d2d6d3f" containerName="kube-rbac-proxy"
Jul 20 10:49:03.614693 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.614676    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.617292 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.617269    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7trj\"/\"kube-root-ca.crt\""
Jul 20 10:49:03.617405 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.617384    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f7trj\"/\"openshift-service-ca.crt\""
Jul 20 10:49:03.618734 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.618689    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f7trj\"/\"default-dockercfg-rjvgf\""
Jul 20 10:49:03.620722 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.620686    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:03.620988 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.620966    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7trj/must-gather-xhtp4"]
Jul 20 10:49:03.704019 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.703985    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/411cdf80-495f-4c69-baee-f164047765ae-must-gather-output\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.704199 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.704059    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2gx\" (UniqueName: \"kubernetes.io/projected/411cdf80-495f-4c69-baee-f164047765ae-kube-api-access-zc2gx\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.720661 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.720634    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:03.804777 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.804702    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/411cdf80-495f-4c69-baee-f164047765ae-must-gather-output\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.804777 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.804769    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2gx\" (UniqueName: \"kubernetes.io/projected/411cdf80-495f-4c69-baee-f164047765ae-kube-api-access-zc2gx\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.805100 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.805016    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/411cdf80-495f-4c69-baee-f164047765ae-must-gather-output\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.812752 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.812727    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2gx\" (UniqueName: \"kubernetes.io/projected/411cdf80-495f-4c69-baee-f164047765ae-kube-api-access-zc2gx\") pod \"must-gather-xhtp4\" (UID: \"411cdf80-495f-4c69-baee-f164047765ae\") " pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:03.924874 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:03.924841    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7trj/must-gather-xhtp4"
Jul 20 10:49:04.053653 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:04.053617    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7trj/must-gather-xhtp4"]
Jul 20 10:49:04.056926 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:49:04.056863    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod411cdf80_495f_4c69_baee_f164047765ae.slice/crio-4696e1736663c0e23ad6e0c676f8d89127c5e1179745f0f0a1feaf99afb46881 WatchSource:0}: Error finding container 4696e1736663c0e23ad6e0c676f8d89127c5e1179745f0f0a1feaf99afb46881: Status 404 returned error can't find the container with id 4696e1736663c0e23ad6e0c676f8d89127c5e1179745f0f0a1feaf99afb46881
Jul 20 10:49:04.058695 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:04.058676    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 10:49:04.915203 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:04.915172    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7trj/must-gather-xhtp4" event={"ID":"411cdf80-495f-4c69-baee-f164047765ae","Type":"ContainerStarted","Data":"521c8fb2390907399446a0661e08ce5a780315b86cb864c8f99d9014d4f248e0"}
Jul 20 10:49:04.915515 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:04.915224    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7trj/must-gather-xhtp4" event={"ID":"411cdf80-495f-4c69-baee-f164047765ae","Type":"ContainerStarted","Data":"4696e1736663c0e23ad6e0c676f8d89127c5e1179745f0f0a1feaf99afb46881"}
Jul 20 10:49:05.924324 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:05.924280    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7trj/must-gather-xhtp4" event={"ID":"411cdf80-495f-4c69-baee-f164047765ae","Type":"ContainerStarted","Data":"6f6a6419cd1604e6dcb114b93acb64bb7a41ac2cee2b536a7e753ff901b623b4"}
Jul 20 10:49:05.941591 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:05.941531    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f7trj/must-gather-xhtp4" podStartSLOduration=2.189507948 podStartE2EDuration="2.94151544s" totalImagesPullingTime="752.007492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 10:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:49:05.940020169 +0000 UTC m=+3129.987444914" watchObservedRunningTime="2026-07-20 10:49:05.94151544 +0000 UTC m=+3129.988940181"
Jul 20 10:49:06.209215 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:06.209140    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q7gmb_f4142bda-92f5-4056-a55e-7d4bd77cd619/global-pull-secret-syncer/0.log"
Jul 20 10:49:06.299815 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:06.299787    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log"
Jul 20 10:49:06.430509 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:06.430478    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-155.ec2.internal_48c98dd19a1c52c34781facdd3c26946/haproxy/0.log"
Jul 20 10:49:09.021717 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:09.021682    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:09.088982 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:09.088910    2546 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4q64h_1ab90c7c-db4f-4f60-9bc5-fe643ce16a9a/konnectivity-agent/0.log" limit=1024
Jul 20 10:49:10.484806 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.484774    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/alertmanager/0.log"
Jul 20 10:49:10.528051 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.528015    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/config-reloader/0.log"
Jul 20 10:49:10.573965 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.573935    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/kube-rbac-proxy-web/0.log"
Jul 20 10:49:10.616861 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.616800    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/kube-rbac-proxy/0.log"
Jul 20 10:49:10.653983 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.653928    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/kube-rbac-proxy-metric/0.log"
Jul 20 10:49:10.677775 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.677705    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/prom-label-proxy/0.log"
Jul 20 10:49:10.702386 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.702357    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7dfd8a09-9c00-4af8-bde3-205b318447f5/init-config-reloader/0.log"
Jul 20 10:49:10.756125 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.756029    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/1.log"
Jul 20 10:49:10.867962 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.867926    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-m5ggr_7be60d37-9c23-4eb3-bfcc-c5688f5ee538/cluster-monitoring-operator/0.log"
Jul 20 10:49:10.979909 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:10.979869    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75b788647d-bb5bd_ba6b2010-37a4-4601-9589-b5710893e6de/metrics-server/0.log"
Jul 20 10:49:11.107912 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.107883    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xqsv_2e62c52e-f97c-4dd0-8885-8a8aa90eb536/node-exporter/0.log"
Jul 20 10:49:11.133152 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.133119    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xqsv_2e62c52e-f97c-4dd0-8885-8a8aa90eb536/kube-rbac-proxy/0.log"
Jul 20 10:49:11.155927 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.155902    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xqsv_2e62c52e-f97c-4dd0-8885-8a8aa90eb536/init-textfile/0.log"
Jul 20 10:49:11.565016 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.564911    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-nxr76_846d321e-a71e-4543-9695-10de6cf5e01e/prometheus-operator-admission-webhook/0.log"
Jul 20 10:49:11.599957 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.599922    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8467c64b86-hfwvj_9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef/telemeter-client/0.log"
Jul 20 10:49:11.626028 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.625998    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8467c64b86-hfwvj_9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef/reload/0.log"
Jul 20 10:49:11.645850 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.645822    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8467c64b86-hfwvj_9d2adc5a-9e60-4ce0-a0c8-fd63f0f2d5ef/kube-rbac-proxy/0.log"
Jul 20 10:49:11.680815 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.680787    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/thanos-query/0.log"
Jul 20 10:49:11.702885 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.702857    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/kube-rbac-proxy-web/0.log"
Jul 20 10:49:11.726835 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.726804    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/kube-rbac-proxy/0.log"
Jul 20 10:49:11.752343 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.752303    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/prom-label-proxy/0.log"
Jul 20 10:49:11.777536 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.777507    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/kube-rbac-proxy-rules/0.log"
Jul 20 10:49:11.809136 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:11.809110    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76cf56f478-zqrdd_58098f4e-dd1a-480b-8c8d-48a83c40eb00/kube-rbac-proxy-metrics/0.log"
Jul 20 10:49:13.099489 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:13.099466    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-5s7hz_35ffa5a5-c4eb-4dfa-b27b-af607712fe9a/networking-console-plugin/0.log"
Jul 20 10:49:13.560335 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:13.560264    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/1.log"
Jul 20 10:49:13.565959 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:13.565931    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-pcqb5_19a58fee-11c7-436b-8da3-648634b1f44c/console-operator/2.log"
Jul 20 10:49:13.934867 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:13.934834    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-795b5d4c5f-58cds_59bdb034-d7d9-4494-9a46-7967e6e17099/console/0.log"
Jul 20 10:49:13.972541 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:13.972501    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-ft9gm_268299b4-1d04-4c04-a83b-5557ffc339df/download-server/0.log"
Jul 20 10:49:14.250777 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.250681    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"]
Jul 20 10:49:14.256586 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.256560    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.263915 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.263890    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"]
Jul 20 10:49:14.309698 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.309661    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-podres\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.309698 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.309701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-sys\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.309890 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.309803    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-proc\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.309927 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.309903    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptxq\" (UniqueName: \"kubernetes.io/projected/a415638f-e044-4ff6-b237-b5eb617265ca-kube-api-access-lptxq\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.309962 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.309931    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-lib-modules\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410512 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410476    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-podres\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410527    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-sys\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410577    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-proc\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410607    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptxq\" (UniqueName: \"kubernetes.io/projected/a415638f-e044-4ff6-b237-b5eb617265ca-kube-api-access-lptxq\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410692 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410670    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-podres\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410674    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-sys\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410677    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-proc\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410722    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-lib-modules\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.410913 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.410825    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a415638f-e044-4ff6-b237-b5eb617265ca-lib-modules\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.419823 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.419789    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptxq\" (UniqueName: \"kubernetes.io/projected/a415638f-e044-4ff6-b237-b5eb617265ca-kube-api-access-lptxq\") pod \"perf-node-gather-daemonset-cb5r5\" (UID: \"a415638f-e044-4ff6-b237-b5eb617265ca\") " pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.489658 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.489616    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-phmdw_294fe74b-4e25-4eff-bd9b-c0bff79829a9/volume-data-source-validator/0.log"
Jul 20 10:49:14.572169 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.572057    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:14.739413 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.739390    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"]
Jul 20 10:49:14.742046 ip-10-0-143-155 kubenswrapper[2546]: W0720 10:49:14.742000    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda415638f_e044_4ff6_b237_b5eb617265ca.slice/crio-35ce3040c89896a5997eab75be9ca6aa3b780e80df6f4f92cb5fad4a4db695ce WatchSource:0}: Error finding container 35ce3040c89896a5997eab75be9ca6aa3b780e80df6f4f92cb5fad4a4db695ce: Status 404 returned error can't find the container with id 35ce3040c89896a5997eab75be9ca6aa3b780e80df6f4f92cb5fad4a4db695ce
Jul 20 10:49:14.986377 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.986334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5" event={"ID":"a415638f-e044-4ff6-b237-b5eb617265ca","Type":"ContainerStarted","Data":"e4f05a25622e0a6b3a45d3e5df96c3cee17f90796a8957922efabf2458f79555"}
Jul 20 10:49:14.986555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.986404    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5" event={"ID":"a415638f-e044-4ff6-b237-b5eb617265ca","Type":"ContainerStarted","Data":"35ce3040c89896a5997eab75be9ca6aa3b780e80df6f4f92cb5fad4a4db695ce"}
Jul 20 10:49:14.986555 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:14.986449    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:15.007506 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:15.007452    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5" podStartSLOduration=1.007437433 podStartE2EDuration="1.007437433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 10:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 10:49:15.004637993 +0000 UTC m=+3139.052062749" watchObservedRunningTime="2026-07-20 10:49:15.007437433 +0000 UTC m=+3139.054862177"
Jul 20 10:49:15.431711 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:15.431674    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j7v_2eb3e353-a052-4603-917b-754e8fef71b2/dns/0.log"
Jul 20 10:49:15.457858 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:15.457830    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j7v_2eb3e353-a052-4603-917b-754e8fef71b2/kube-rbac-proxy/0.log"
Jul 20 10:49:15.529242 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:15.529211    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4l429_df265ea5-1894-4478-8759-17110f267128/dns-node-resolver/0.log"
Jul 20 10:49:16.009381 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:16.009352    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-22smx_742d5ccd-da6c-4f25-b376-9d9b85e9cee1/node-ca/0.log"
Jul 20 10:49:17.194995 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.194965    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xwfhn_47a080b5-3e17-4b8f-8f0c-b35f5655bde7/serve-healthcheck-canary/0.log"
Jul 20 10:49:17.532456 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.532359    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-c54fp_2b0979e2-1e37-4d9a-a017-3771bf4a5d27/insights-operator/0.log"
Jul 20 10:49:17.533773 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.533751    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-c54fp_2b0979e2-1e37-4d9a-a017-3771bf4a5d27/insights-operator/1.log"
Jul 20 10:49:17.617431 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.617403    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npw5t_94a1b7fa-4991-419f-9bdd-6f474c315059/kube-rbac-proxy/0.log"
Jul 20 10:49:17.640206 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.640183    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npw5t_94a1b7fa-4991-419f-9bdd-6f474c315059/exporter/0.log"
Jul 20 10:49:17.659240 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:17.659211    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-npw5t_94a1b7fa-4991-419f-9bdd-6f474c315059/extractor/0.log"
Jul 20 10:49:19.853655 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:19.853572    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-645f4f9567-r96hp_72d95521-b827-46c9-aefd-2f6d411d44e2/manager/0.log"
Jul 20 10:49:20.007590 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:20.007567    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-cq2s7_d3e74f7f-b633-453a-aebe-c55efba6e9cd/manager/0.log"
Jul 20 10:49:20.074676 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:20.074651    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nml64_d4ae3989-afef-4444-86a0-c4f27df04db5/seaweedfs/0.log"
Jul 20 10:49:21.002284 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:21.002254    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f7trj/perf-node-gather-daemonset-cb5r5"
Jul 20 10:49:25.223977 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.223928    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-27ggd_92cf6d64-b819-4546-8442-039af4c1b4c9/kube-multus/0.log"
Jul 20 10:49:25.276435 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.276403    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/kube-multus-additional-cni-plugins/0.log"
Jul 20 10:49:25.299679 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.299650    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/egress-router-binary-copy/0.log"
Jul 20 10:49:25.323261 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.323236    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/cni-plugins/0.log"
Jul 20 10:49:25.351056 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.351032    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/bond-cni-plugin/0.log"
Jul 20 10:49:25.373168 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.373141    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/routeoverride-cni/0.log"
Jul 20 10:49:25.396417 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.396396    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/whereabouts-cni-bincopy/0.log"
Jul 20 10:49:25.419983 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.419900    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ctm7j_2578aede-60eb-4f71-94ed-06e90d6cf997/whereabouts-cni/0.log"
Jul 20 10:49:25.779191 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.779129    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qzr2_44618c9c-a6c5-4e94-aedf-cadacba06315/network-metrics-daemon/0.log"
Jul 20 10:49:25.796851 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:25.796831    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5qzr2_44618c9c-a6c5-4e94-aedf-cadacba06315/kube-rbac-proxy/0.log"
Jul 20 10:49:26.603572 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.603543    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/ovn-controller/0.log"
Jul 20 10:49:26.618949 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.618921    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/ovn-acl-logging/0.log"
Jul 20 10:49:26.640155 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.640135    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/kube-rbac-proxy-node/0.log"
Jul 20 10:49:26.669277 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.669257    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 20 10:49:26.683634 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.683617    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/northd/0.log"
Jul 20 10:49:26.702711 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.702695    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/nbdb/0.log"
Jul 20 10:49:26.722233 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.722217    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/sbdb/0.log"
Jul 20 10:49:26.779584 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:26.779556    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cqlg8_72706163-7492-4705-9a19-452370fc456b/ovnkube-controller/0.log"
Jul 20 10:49:28.116886 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:28.116860    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7bc5b6bf85-wkvxw_66a2f67e-595b-4feb-8250-0c2ee8467e32/check-endpoints/0.log"
Jul 20 10:49:28.138515 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:28.138486    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5d4qt_c40010c2-e8eb-498e-94ae-e59f52a4fa6f/network-check-target-container/0.log"
Jul 20 10:49:28.997328 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:28.997300    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rsgch_17948cd8-1721-4f40-8576-c21cda2c71c6/iptables-alerter/0.log"
Jul 20 10:49:29.576105 ip-10-0-143-155 kubenswrapper[2546]: I0720 10:49:29.576080    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8q2vn_6a62ec41-3d0d-462d-a0e8-fd55f5b3b565/tuned/0.log"