Jul 28 17:10:19.328294 ip-10-0-136-80 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:19.730577 ip-10-0-136-80 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:19.730577 ip-10-0-136-80 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:19.730577 ip-10-0-136-80 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:19.730577 ip-10-0-136-80 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:19.730577 ip-10-0-136-80 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:19.733606 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.733524    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:19.737727 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737713    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.737727 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737727    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737731    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737736    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737739    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737742    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737745    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737748    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737750    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737753    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737756    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737759    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737762    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737764    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737767    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737769    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737772    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737775    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737777    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737780    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737784    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.737791 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737786    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737789    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737791    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737794    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737797    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737800    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737803    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737805    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737808    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737811    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737813    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737816    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737819    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737822    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737824    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737827    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737829    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737832    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737835    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737837    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.738312 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737840    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737845    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737847    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737850    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737852    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737855    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737857    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737860    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737862    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737865    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737867    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737870    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737873    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737875    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737880    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737885    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737888    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737891    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737894    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.738793 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737897    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737900    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737903    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737905    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737908    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737910    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737913    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737916    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737918    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737921    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737923    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737926    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737928    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737931    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737933    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737936    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737939    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737941    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737944    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737946    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.739276 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737949    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737951    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737954    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737956    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737959    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737961    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737964    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737968    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737971    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737976    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737979    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737982    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737985    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.737999    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738003    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738005    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738009    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738013    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738017    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.739749 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738725    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738731    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738734    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738737    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738740    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738742    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738745    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738749    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738752    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738755    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738758    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738761    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738763    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738766    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738768    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738771    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738773    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738776    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738779    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.740233 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738782    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738785    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738788    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738790    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738793    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738796    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738799    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738802    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738805    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738807    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738810    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738812    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738815    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738818    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738820    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738823    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738825    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738828    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738831    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738833    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.740735 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738837    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738841    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738844    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738847    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738850    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738853    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738856    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738859    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738862    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738865    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738867    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738870    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738873    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738876    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738879    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738882    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738885    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738887    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738889    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.741250 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738892    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738895    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738897    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738900    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738902    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738905    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738907    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738910    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738912    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738915    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738917    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738920    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738922    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738925    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738928    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738930    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738933    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738936    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738938    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738941    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.741716 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738943    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738946    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738948    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738951    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738953    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738956    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738961    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738965    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738968    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738971    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738974    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738976    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738979    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738981    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.738984    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739003    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739007    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739011    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739015    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739019    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.742237 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739023    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739415    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739420    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739423    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739426    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739429    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739432    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739434    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739437    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739439    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739442    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739445    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739448    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739450    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739452    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739455    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739459    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739462    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739464    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739467    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.742728 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739475    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739478    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739480    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739483    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739485    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739488    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739491    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739493    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739496    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739498    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739501    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739503    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739506    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739509    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739511    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739514    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739516    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739519    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739521    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.743251 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739524    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739528    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739532    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739534    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739537    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739539    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739542    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739544    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739547    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739549    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739552    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739555    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739557    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739560    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739567    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739570    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739573    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739576    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739578    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739581    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.743714 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739583    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739586    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739588    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739591    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739593    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739596    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739598    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739601    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739604    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739606    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739609    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739611    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739615    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739619    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739622    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739625    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739627    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739630    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739632    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739636    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.744231 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739638    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739643    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739646    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739648    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739651    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739653    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739656    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739664    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739667    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739669    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739672    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739674    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739677    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739680    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739682    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739684    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739692    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739695    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739698    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.744713 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739701    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.739704    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739778    2552 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739789    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739802    2552 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739809    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739815    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739820    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739824    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739828    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739831    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739834    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739838    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739841    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739847    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739850    2552 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739853    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739856    2552 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739859    2552 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739862    2552 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739872    2552 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739880    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739884    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:19.745183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739887    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739890    2552 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739894    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739897    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739900    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739903    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739907    2552 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739910    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739913    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739916    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739919    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739923    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739926    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739929    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739932    2552 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739935    2552 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739938    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739944    2552 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739947    2552 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739950    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739953    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739956    2552 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739960    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739964    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739967    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:19.745734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739970    2552 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739973    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739975    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739978    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739981    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.739984    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740006    2552 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740009    2552 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740015    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740018    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740021    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740025    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740028    2552 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740033    2552 flags.go:64] FLAG: --help="false"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740036    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740039    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740042    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740045    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740048    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740051    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740054    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740057    2552 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740060    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:19.746370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740063    2552 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740066    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740068    2552 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740071    2552 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740074    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740077    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740080    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740084    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740087    2552 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740090    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740093    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740096    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740101    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740104    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740107    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740110    2552 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740118    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740121    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740124    2552 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740127    2552 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740131    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740134    2552 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740149    2552 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740152    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740155    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:19.747103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740159    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740162    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740165    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740167    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740170    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740177    2552 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740180    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740183    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740186    2552 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740189    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740194    2552 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740196    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740199    2552 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740202    2552 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740205    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740209    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af767041e32ca915"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740213    2552 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740216    2552 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740219    2552 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740222    2552 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740225    2552 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740228    2552 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740231    2552 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740234    2552 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740244    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:19.748021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740247    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740249    2552 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740252    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740255    2552 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740259    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740262    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740265    2552 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740268    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740271    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740274    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740277    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740279    2552 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740282    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740285    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740288    2552 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740291    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740294    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740297    2552 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740299    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740308    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740311    2552 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740313    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740321    2552 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740324    2552 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740326    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:19.748655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740329    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740332    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740335    2552 flags.go:64] FLAG: --v="2"
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740339    2552 flags.go:64] FLAG: --version="false"
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740343    2552 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740347    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740350    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740508    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740513    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740518    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740522    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740527    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740530    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740533    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740536    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740539    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740542    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740544    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740547    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740550    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740552    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.749283 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740555    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740557    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740560    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740562    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740565    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740568    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740570    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740573    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740577    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740579    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740582    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740584    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740587    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740589    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740592    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740594    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740597    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740600    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740602    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740605    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.749792 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740608    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740611    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740615    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740617    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740620    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740622    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740625    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740627    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740630    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740632    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740635    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740638    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740641    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740643    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740646    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740648    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740651    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740653    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740657    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740660    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.750330 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740665    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740667    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740670    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740673    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740676    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740679    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740681    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740684    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740686    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740689    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740692    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740694    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740698    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740700    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740704    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740707    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740709    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740712    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740715    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740718    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.750822 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740720    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740723    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740725    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740728    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740730    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740732    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740735    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740738    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740740    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740743    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740745    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740748    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740752    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740754    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740757    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740759    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740762    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740764    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740767    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.751332 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740769    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740772    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740774    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740777    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740779    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.740781    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.740793    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.748831    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.748845    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.748864    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.748870    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748924    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748928    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748931    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748935    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.751797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748937    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748940    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748943    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748946    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748949    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748952    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748955    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748958    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748961    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748963    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748966    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748969    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748972    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748974    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748977    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748979    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748982    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.748985    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749006    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.752195 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749010    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749014    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749017    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749020    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749022    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749025    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749028    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749031    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749033    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749035    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749038    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749042    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749045    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749048    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749051    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749054    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749056    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749059    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749062    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.752662 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749065    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749067    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749070    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749073    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749075    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749078    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749081    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749084    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749086    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749089    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749091    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749093    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749096    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749099    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749101    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749104    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749106    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749109    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749112    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749114    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.753211 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749117    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749119    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749122    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749124    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749127    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749130    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749132    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749135    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749137    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749139    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749142    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749145    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749147    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749150    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749152    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749155    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749158    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749161    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749164    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749166    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.753819 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749169    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749173    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749176    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749179    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749182    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749184    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749187    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749189    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749192    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749194    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749197    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749199    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749202    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749204    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749207    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749209    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.754313 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749212    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.749217    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749349    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749355    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749358    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749362    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749367    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749370    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749373    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749377    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749381    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749384    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749387    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749389    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:19.754691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749393    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749396    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749399    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749402    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749404    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749407    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749410    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749413    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749415    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749418    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749421    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749424    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749427    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749429    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749432    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749435    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749437    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749441    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749444    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749446    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:19.755049 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749449    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749451    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749454    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749456    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749459    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749461    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749464    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749467    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749469    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749472    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749474    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749477    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749479    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749482    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749485    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749487    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749490    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749492    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749495    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:19.755534 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749497    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749500    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749502    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749505    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749508    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749510    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749513    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749515    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749518    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749520    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749524    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749527    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749530    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749533    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749535    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749538    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749540    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749543    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749545    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:19.756005 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749548    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749550    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749553    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749555    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749557    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749560    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749563    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749566    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749569    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749572    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749574    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749577    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749579    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749582    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749584    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749586    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749589    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749592    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749594    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749597    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:19.756497 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749600    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749602    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749604    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749607    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749610    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749612    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749615    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749617    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:19.749620    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.749624    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.749721    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.752393    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.753608    2552 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.753708    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:19.756969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.753775    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:19.757359 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.756169    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:19.757393 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.757354    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:19.771998 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.771971    2552 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:19.774986 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.774973    2552 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:19.775717 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.775705    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:19.778050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.778026    2552 fs.go:135] Filesystem UUIDs: map[38200284-3eff-45f0-97a6-73d35c3321a9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8c6193b2-fb8d-48d1-9522-bfcbe9b6cedd:/dev/nvme0n1p4]
Jul 28 17:10:19.778114 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.778050    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:19.779452 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.779435    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:19.784056 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.783934    2552 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:19.782647344 +0000 UTC m=+0.334957227 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100166 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bc989def288a4a35d518df3edb05c SystemUUID:ec2bc989-def2-88a4-a35d-518df3edb05c BootID:b3aa4178-7fe2-4308-b129-4b749de68ec0 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6f:f6:bd:9c:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6f:f6:bd:9c:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:87:c7:a2:a0:f9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:19.784056 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.784049    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:19.784177 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.784126    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:19.787592 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.787570    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:19.787739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.787594    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:19.788426 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.788416    2552 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:19.788463 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.788429    2552 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:19.788463 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.788437    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:19.789229 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.789218    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:19.790073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.790064    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:19.791764 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.791754    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:19.791884 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.791875    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:19.791918 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.791902    2552 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:19.791949 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.791918    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:19.791949 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.791931    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:19.792009 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.792000    2552 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:19.792042 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.792010    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:19.793948 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.793899    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:19.794051 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.793982    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:19.796782 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.796762    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:19.798907 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.798893    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:19.798964 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.798923    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:19.800311 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800299    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800317    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800323    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800328    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800337    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800342    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800349    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:19.800352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800354    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:19.800542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800361    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:19.800542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800366    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:19.800542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800374    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:19.800542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.800379    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:19.801741 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.801726    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:19.801804 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.801743    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:19.802161 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.802140    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27qb8"
Jul 28 17:10:19.803191 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.803164    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:19.803253 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.803224    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:19.805098 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.805086    2552 server.go:1268] "Started kubelet"
Jul 28 17:10:19.805237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.805186    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:19.805330 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.805255    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:19.805801 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.805769    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:19.805839 ip-10-0-136-80 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:19.807175 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.807159    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:19.808479 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.807462    2552 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:19.811509 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.811493    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27qb8"
Jul 28 17:10:19.811735 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.811717    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:19.814560 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.814529    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:19.816671 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.814632    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-80.ec2.internal.18c6834189af4ded  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-80.ec2.internal,UID:ip-10-0-136-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-80.ec2.internal,},FirstTimestamp:2026-07-28 17:10:19.805060589 +0000 UTC m=+0.357370470,LastTimestamp:2026-07-28 17:10:19.805060589 +0000 UTC m=+0.357370470,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-80.ec2.internal,}"
Jul 28 17:10:19.816947 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.816926    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:19.816947 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.816942    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:19.817272 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.817018    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:19.817598 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817579    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:19.817598 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817583    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:19.817744 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817626    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:19.817744 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817689    2552 factory.go:55] Registering systemd factory
Jul 28 17:10:19.817744 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817703    2552 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:19.817928 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817913    2552 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:19.817928 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817928    2552 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:19.818040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817932    2552 factory.go:153] Registering CRI-O factory
Jul 28 17:10:19.818040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.817947    2552 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:19.818040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.818028    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:19.818220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.818054    2552 factory.go:103] Registering Raw factory
Jul 28 17:10:19.818220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.818070    2552 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:19.818551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.818525    2552 manager.go:349] Starting recovery of all containers
Jul 28 17:10:19.821404 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.821386    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:19.829885 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.829860    2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-80.ec2.internal\" not found" node="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:19.832974 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.832959    2552 manager.go:354] Recovery completed
Jul 28 17:10:19.834560 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.834543    2552 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 17:10:19.837647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.837632    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:19.839503 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.839486    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:19.839564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.839519    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:19.839564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.839534    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:19.840036 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.840021    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:19.840036 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.840036    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:19.840136 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.840052    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:19.840136 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.840056    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:19.841118 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841103    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:19.841191 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841129    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:19.841191 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841139    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:19.841697 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841684    2552 policy_none.go:49] "None policy: Start"
Jul 28 17:10:19.841754 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841704    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:19.841754 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.841718    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:19.842754 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.842742    2552 policy_none.go:47] "Start"
Jul 28 17:10:19.918333 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.918311    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:19.965113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.965086    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:19.966302 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.966285    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:19.966369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.966323    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:19.966369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.966346    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:19.966469 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.966379    2552 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 28 17:10:19.969869 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.969845    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:19.999809 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.999633    2552 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:19.999904 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:19.999835    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:19.999904 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:19.999850    2552 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:20.000134 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.000120    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:20.000216 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.000136    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:20.000267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.000228    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:20.000339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.000324    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:20.000339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.000339    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:20.000866 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.000846    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:20.000933 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.000891    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.066973 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.066944    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"]
Jul 28 17:10:20.067081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.067051    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:20.068606 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.068590    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:20.068698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.068621    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:20.068698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.068634    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:20.069894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.069880    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:20.070040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070028    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.070091 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070063    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:20.070647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070632    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:20.070724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070657    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:20.070724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070670    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:20.070724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070678    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:20.070724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070701    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:20.070724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.070712    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:20.071972 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.071955    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.072070 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.071987    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:20.073592 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.073576    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:20.073667 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.073602    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:20.073667 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.073611    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:20.099557 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.099538    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-80.ec2.internal\" not found" node="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.100357 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.100342    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:20.101196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.101182    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:20.101267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.101205    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:20.101267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.101225    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:20.101267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.101242    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.103948 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.103933    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-80.ec2.internal\" not found" node="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.113041 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.113025    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.113109 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.113043    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-80.ec2.internal\": node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.119543 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.119528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.119617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.119549    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.119617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.119575    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb9f2dabc625c037e2ff47a5be458f87-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"eb9f2dabc625c037e2ff47a5be458f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.162222 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.162201    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.220176 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220152    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb9f2dabc625c037e2ff47a5be458f87-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"eb9f2dabc625c037e2ff47a5be458f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.220264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220146    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb9f2dabc625c037e2ff47a5be458f87-config\") pod \"kube-apiserver-proxy-ip-10-0-136-80.ec2.internal\" (UID: \"eb9f2dabc625c037e2ff47a5be458f87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.220264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220220    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.220264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220233    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.220264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220258    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.220407 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.220275    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1461c37f00540e8e6e35dbf0066c0e3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal\" (UID: \"e1461c37f00540e8e6e35dbf0066c0e3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.262303 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.262245    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.362684 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.362662    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.402962 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.402944    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.407945 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.407924    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.463652 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.463626    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.564146 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.564087    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.664532 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.664491    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-80.ec2.internal\" not found"
Jul 28 17:10:20.694851 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.694831    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:20.718308 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.718283    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.738856 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.738833    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:20.741268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.741254    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.753449 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.753429    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:20.753556 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.753538    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:20.753614 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.753584    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:20.753614 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.753577    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:20.753695 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.753618    2552 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a7a7e4541a572499d8459b42f1b49b85-570823774f58758a.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.136.80:41254->44.205.236.177:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal"
Jul 28 17:10:20.792299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.792274    2552 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:20.816287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.816223    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:20 +0000 UTC" deadline="2028-04-26 03:39:38.715165242 +0000 UTC"
Jul 28 17:10:20.816287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.816251    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15298h29m17.898918048s"
Jul 28 17:10:20.816682 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.816666    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:20.819001 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.817165    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:20.820797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.820768    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-g4wzj","openshift-network-operator/iptables-alerter-kq57b","openshift-ovn-kubernetes/ovnkube-node-h6lk7","kube-system/global-pull-secret-syncer-pdtks","kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal","openshift-dns/node-resolver-dx6mc","openshift-image-registry/node-ca-l8bj2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal","openshift-multus/multus-additional-cni-plugins-zblqh","openshift-multus/multus-pscn6","openshift-multus/network-metrics-daemon-dww89","kube-system/konnectivity-agent-9kq74","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl","openshift-cluster-node-tuning-operator/tuned-rzck2"]
Jul 28 17:10:20.822333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.822317    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:20.823661 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.823635    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:20.823756 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.823718    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:20.825168 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.825149    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.826017 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.826000    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.826017 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.826013    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.826224 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.826004    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m5cqz\""
Jul 28 17:10:20.826224 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.826008    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:20.826411 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.826396    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:20.826502 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.826480    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:20.827677 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.827664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.829145 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.829125    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:20.830453 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830433    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.830579 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830558    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6vjvp\""
Jul 28 17:10:20.830651 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830610    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.830710 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830611    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:20.830789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830769    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:20.830959 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.830941    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:20.831546 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.831529    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:20.831546 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.831540    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.831634 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.831597    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.831953 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.831938    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.832331 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.832312    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.832425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.832357    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.832425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.832362    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.832425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.832400    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xh9rt\""
Jul 28 17:10:20.832655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.832641    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prhxd\""
Jul 28 17:10:20.833432 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.833413    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:20.833534 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.833521    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:20.833692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.833675    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.834071 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.834048    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.834308 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.834289    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:20.834387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.834319    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:20.835115 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.835097    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tzw6b\""
Jul 28 17:10:20.835199 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.835102    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.835696 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.835678    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:20.835750 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.835683    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.835785 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.835747    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wkzrz\""
Jul 28 17:10:20.836526 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.836507    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.836623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.836541    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:20.836623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.836551    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:20.836623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.836611    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:20.836779 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.836627    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:20.836779 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:20.836691    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:20.837191 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.837174    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:20.837275 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.837205    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jz6bz\""
Jul 28 17:10:20.837917 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.837901    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:20.838073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.838051    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-r7w8h\""
Jul 28 17:10:20.838354 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.838339    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:20.841337 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.841317    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:20.841459 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.841434    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:20.841565 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.841474    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wd6mf\""
Jul 28 17:10:20.874743 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.874719    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwf8g"
Jul 28 17:10:20.883328 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.883313    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xwf8g"
Jul 28 17:10:20.889250 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.889233    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:20.918643 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.918626    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:20.924940 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.924922    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cni-binary-copy\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925023 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.924948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-system-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925023 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.924969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-multus\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925025    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-var-lib-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925056    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-node-log\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925087    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovn-node-metrics-cert\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-script-lib\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925121    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-systemd\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925161    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9fq\" (UniqueName: \"kubernetes.io/projected/069a7e39-3dc9-498d-80c2-b0ef2e353733-kube-api-access-ft9fq\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925182    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-socket-dir-parent\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925199    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-k8s-cni-cncf-io\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925220    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-dbus\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925242    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925258    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s78l\" (UniqueName: \"kubernetes.io/projected/09870d81-7076-49f2-a276-6530e16cf472-kube-api-access-6s78l\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925284    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925319    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-kubelet\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-kubelet-config\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-registration-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.925382 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmc2\" (UniqueName: \"kubernetes.io/projected/1af24ccc-78ed-445a-891a-bf25f722e76a-kube-api-access-lwmc2\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925391    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-host\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925415    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtcc\" (UniqueName: \"kubernetes.io/projected/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-kube-api-access-vrtcc\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925430    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-ovn\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925448    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-config\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925476    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysconfig\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925512    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-conf\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925537    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-var-lib-kubelet\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925554    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15e89d16-f69e-458d-848a-53e2a1dfbcdd-iptables-alerter-script\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925571    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-device-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.925589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925587    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c14d132-50ec-45ad-9967-87f75f5d4730-agent-certs\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925617    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925654    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cnibin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925681    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-kubernetes\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-sys\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-tuned\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925737    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-tmp\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925752    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-etc-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925767    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2k6x\" (UniqueName: \"kubernetes.io/projected/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-kube-api-access-z2k6x\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925782    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-slash\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925797    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-log-socket\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-system-cni-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.925880 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-os-release\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-host\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925910    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925925    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-netd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925946    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-os-release\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdnj\" (UniqueName: \"kubernetes.io/projected/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-kube-api-access-8xdnj\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-socket-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.925992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-serviceca\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-etc-kubernetes\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926053    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-lib-modules\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926081    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-multus-certs\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-systemd-units\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926126    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbt8\" (UniqueName: \"kubernetes.io/projected/52ed84b2-9963-498c-8594-54462ee49011-kube-api-access-rhbt8\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-env-overrides\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926158    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-run\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.926320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e89d16-f69e-458d-848a-53e2a1dfbcdd-host-slash\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926194    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-daemon-config\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-kubelet\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926230    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gm6h\" (UniqueName: \"kubernetes.io/projected/8bf73946-55d9-4630-8bc2-f6c27634a26c-kube-api-access-5gm6h\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926263    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cnibin\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926283    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-binary-copy\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926298    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926328    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-modprobe-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926368    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-bin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926405    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09870d81-7076-49f2-a276-6530e16cf472-tmp-dir\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926436    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926469    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-etc-selinux\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-conf-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926522    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-netns\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926546    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-bin\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.926957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926569    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09870d81-7076-49f2-a276-6530e16cf472-hosts-file\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926586    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926604    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-sys-fs\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926621    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-netns\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926636    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-systemd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926665    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926704    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq5f\" (UniqueName: \"kubernetes.io/projected/15e89d16-f69e-458d-848a-53e2a1dfbcdd-kube-api-access-6jq5f\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926729    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-hostroot\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926753    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c14d132-50ec-45ad-9967-87f75f5d4730-konnectivity-ca\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:20.927518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:20.926780    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:21.027437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027403    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.027437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027434    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-system-cni-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027452    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-os-release\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-host\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027514    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-netd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-system-cni-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-os-release\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027543    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027577    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-os-release\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027585    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-host\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027593    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-netd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027590    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdnj\" (UniqueName: \"kubernetes.io/projected/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-kube-api-access-8xdnj\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027639    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-socket-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027644    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-os-release\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.027695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027668    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-serviceca\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027764    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-etc-kubernetes\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027789    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-socket-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027793    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027845    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-etc-kubernetes\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027853    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-lib-modules\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027885    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-multus-certs\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027916    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-systemd-units\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027940    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-lib-modules\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027943    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbt8\" (UniqueName: \"kubernetes.io/projected/52ed84b2-9963-498c-8594-54462ee49011-kube-api-access-rhbt8\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027972    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-systemd-units\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027974    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-env-overrides\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.027946    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-multus-certs\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028000    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-run\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028041    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e89d16-f69e-458d-848a-53e2a1dfbcdd-host-slash\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028066    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-daemon-config\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-kubelet\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.028485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gm6h\" (UniqueName: \"kubernetes.io/projected/8bf73946-55d9-4630-8bc2-f6c27634a26c-kube-api-access-5gm6h\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028122    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15e89d16-f69e-458d-848a-53e2a1dfbcdd-host-slash\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-run\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028164    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-kubelet\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028199    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cnibin\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028226    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-binary-copy\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028269    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cnibin\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028328    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-serviceca\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028356    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028413    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-modprobe-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028417    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028445    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-bin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028473    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09870d81-7076-49f2-a276-6530e16cf472-tmp-dir\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028483    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-env-overrides\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028513    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-bin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.029153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028539    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-modprobe-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028599    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-etc-selinux\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028615    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-conf-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028632    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-netns\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028657    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-bin\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028673    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-conf-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028681    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09870d81-7076-49f2-a276-6530e16cf472-hosts-file\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028724    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/09870d81-7076-49f2-a276-6530e16cf472-hosts-file\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028717    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028675    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-run-netns\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028740    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-etc-selinux\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028761    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09870d81-7076-49f2-a276-6530e16cf472-tmp-dir\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028763    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-cni-bin\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028744    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-daemon-config\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028801    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-sys-fs\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028813    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.029979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028819    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-binary-copy\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028830    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-netns\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028853    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-sys-fs\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028868    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-systemd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028881    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-netns\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028896    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028911    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-systemd\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028920    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq5f\" (UniqueName: \"kubernetes.io/projected/15e89d16-f69e-458d-848a-53e2a1dfbcdd-kube-api-access-6jq5f\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.028983    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-hostroot\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029018    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-d\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c14d132-50ec-45ad-9967-87f75f5d4730-konnectivity-ca\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029046    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-hostroot\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029059    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029101    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cni-binary-copy\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-system-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029133    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029151    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-multus\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029178    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-var-lib-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.030837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029209    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-system-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029226    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-cni-multus\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029230    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-var-lib-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029250    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-node-log\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.029252    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-node-log\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029280    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovn-node-metrics-cert\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-script-lib\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-systemd\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029367    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9fq\" (UniqueName: \"kubernetes.io/projected/069a7e39-3dc9-498d-80c2-b0ef2e353733-kube-api-access-ft9fq\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029394    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-socket-dir-parent\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.029417    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.529392653 +0000 UTC m=+2.081702546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029439    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-socket-dir-parent\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029450    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-k8s-cni-cncf-io\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-dbus\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029504    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c14d132-50ec-45ad-9967-87f75f5d4730-konnectivity-ca\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029509    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.031455 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029415    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-systemd\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s78l\" (UniqueName: \"kubernetes.io/projected/09870d81-7076-49f2-a276-6530e16cf472-kube-api-access-6s78l\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029566    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-run-k8s-cni-cncf-io\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029582    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-dbus\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-kubelet\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029605    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029621    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-kubelet-config\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029650    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-registration-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029661    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cni-binary-copy\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029678    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmc2\" (UniqueName: \"kubernetes.io/projected/1af24ccc-78ed-445a-891a-bf25f722e76a-kube-api-access-lwmc2\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029680    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-host-var-lib-kubelet\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029737    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-kubelet-config\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029739    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-host\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.029762    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029827    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-registration-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-host\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029853    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtcc\" (UniqueName: \"kubernetes.io/projected/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-kube-api-access-vrtcc\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.031969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029864    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-script-lib\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.029900    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.529863366 +0000 UTC m=+2.082173269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.029948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-ovn\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030004    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-run-ovn\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030010    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-config\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030037    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysconfig\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030060    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-conf\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030087    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-var-lib-kubelet\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030101    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030117    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15e89d16-f69e-458d-848a-53e2a1dfbcdd-iptables-alerter-script\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030131    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysconfig\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-device-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030169    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-var-lib-kubelet\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030190    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c14d132-50ec-45ad-9967-87f75f5d4730-agent-certs\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030204    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-sysctl-conf\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030218    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030223    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1af24ccc-78ed-445a-891a-bf25f722e76a-device-dir\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.032516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cnibin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030262    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-multus-cni-dir\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-kubernetes\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030308    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-cnibin\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-sys\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030341    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-tuned\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030346    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-kubernetes\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030376    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/069a7e39-3dc9-498d-80c2-b0ef2e353733-sys\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-tmp\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-etc-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030437    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k6x\" (UniqueName: \"kubernetes.io/projected/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-kube-api-access-z2k6x\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-slash\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030499    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-log-socket\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030530    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovnkube-config\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030572    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-log-socket\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030587    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-etc-openvswitch\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8bf73946-55d9-4630-8bc2-f6c27634a26c-host-slash\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.030694    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15e89d16-f69e-458d-848a-53e2a1dfbcdd-iptables-alerter-script\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.033699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.033316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-tmp\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.033382    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/069a7e39-3dc9-498d-80c2-b0ef2e353733-etc-tuned\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.033699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.033425    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8bf73946-55d9-4630-8bc2-f6c27634a26c-ovn-node-metrics-cert\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.033699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.033505    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c14d132-50ec-45ad-9967-87f75f5d4730-agent-certs\") pod \"konnectivity-agent-9kq74\" (UID: \"6c14d132-50ec-45ad-9967-87f75f5d4730\") " pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:21.038664 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.038638    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:21.038664 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.038667    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:21.038809 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.038682    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.038809 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.038749    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.538732128 +0000 UTC m=+2.091042021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.039967 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.039940    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdnj\" (UniqueName: \"kubernetes.io/projected/4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963-kube-api-access-8xdnj\") pod \"multus-additional-cni-plugins-zblqh\" (UID: \"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963\") " pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.041675 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.041648    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtcc\" (UniqueName: \"kubernetes.io/projected/b44d39e4-6f0e-43ba-962f-14f740a7bbfe-kube-api-access-vrtcc\") pod \"node-ca-l8bj2\" (UID: \"b44d39e4-6f0e-43ba-962f-14f740a7bbfe\") " pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.042210 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042169    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq5f\" (UniqueName: \"kubernetes.io/projected/15e89d16-f69e-458d-848a-53e2a1dfbcdd-kube-api-access-6jq5f\") pod \"iptables-alerter-kq57b\" (UID: \"15e89d16-f69e-458d-848a-53e2a1dfbcdd\") " pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.042287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042222    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s78l\" (UniqueName: \"kubernetes.io/projected/09870d81-7076-49f2-a276-6530e16cf472-kube-api-access-6s78l\") pod \"node-resolver-dx6mc\" (UID: \"09870d81-7076-49f2-a276-6530e16cf472\") " pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.042287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042224    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gm6h\" (UniqueName: \"kubernetes.io/projected/8bf73946-55d9-4630-8bc2-f6c27634a26c-kube-api-access-5gm6h\") pod \"ovnkube-node-h6lk7\" (UID: \"8bf73946-55d9-4630-8bc2-f6c27634a26c\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.042287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042253    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbt8\" (UniqueName: \"kubernetes.io/projected/52ed84b2-9963-498c-8594-54462ee49011-kube-api-access-rhbt8\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:21.042545 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042523    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmc2\" (UniqueName: \"kubernetes.io/projected/1af24ccc-78ed-445a-891a-bf25f722e76a-kube-api-access-lwmc2\") pod \"aws-ebs-csi-driver-node-slnnl\" (UID: \"1af24ccc-78ed-445a-891a-bf25f722e76a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.042758 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.042741    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9fq\" (UniqueName: \"kubernetes.io/projected/069a7e39-3dc9-498d-80c2-b0ef2e353733-kube-api-access-ft9fq\") pod \"tuned-rzck2\" (UID: \"069a7e39-3dc9-498d-80c2-b0ef2e353733\") " pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.044889 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.044872    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k6x\" (UniqueName: \"kubernetes.io/projected/1df21e2b-1c50-4d77-8985-2aa6b8a75c2a-kube-api-access-z2k6x\") pod \"multus-pscn6\" (UID: \"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a\") " pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.157129 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.157039    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l8bj2"
Jul 28 17:10:21.167255 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.167233    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:21.194234 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.194200    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rzck2"
Jul 28 17:10:21.198314 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.198296    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dx6mc"
Jul 28 17:10:21.204041 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.204015    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zblqh"
Jul 28 17:10:21.216229 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.216186    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069a7e39_3dc9_498d_80c2_b0ef2e353733.slice/crio-f2bce9803c75867516b150e8516df72ecd78144c4e4e1f41a1aa3fdf8e9cbc8e WatchSource:0}: Error finding container f2bce9803c75867516b150e8516df72ecd78144c4e4e1f41a1aa3fdf8e9cbc8e: Status 404 returned error can't find the container with id f2bce9803c75867516b150e8516df72ecd78144c4e4e1f41a1aa3fdf8e9cbc8e
Jul 28 17:10:21.216861 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.216825    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d386b9a_93a0_4b9d_b3cd_1bd3c0e89963.slice/crio-4a79a6db2f35d998100742bf45253c50623cb878ddddc4108db4a154fa96904d WatchSource:0}: Error finding container 4a79a6db2f35d998100742bf45253c50623cb878ddddc4108db4a154fa96904d: Status 404 returned error can't find the container with id 4a79a6db2f35d998100742bf45253c50623cb878ddddc4108db4a154fa96904d
Jul 28 17:10:21.217398 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.217373    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1461c37f00540e8e6e35dbf0066c0e3.slice/crio-817daf4d2847cccaf3b0b959107edd403a700f6bdb1ea1a27687de28e24919cb WatchSource:0}: Error finding container 817daf4d2847cccaf3b0b959107edd403a700f6bdb1ea1a27687de28e24919cb: Status 404 returned error can't find the container with id 817daf4d2847cccaf3b0b959107edd403a700f6bdb1ea1a27687de28e24919cb
Jul 28 17:10:21.218308 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.218280    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9f2dabc625c037e2ff47a5be458f87.slice/crio-3948cc4a934f21005c9c909cf1ed9d0c7ab8168748a123720b4693b8e9a27602 WatchSource:0}: Error finding container 3948cc4a934f21005c9c909cf1ed9d0c7ab8168748a123720b4693b8e9a27602: Status 404 returned error can't find the container with id 3948cc4a934f21005c9c909cf1ed9d0c7ab8168748a123720b4693b8e9a27602
Jul 28 17:10:21.219271 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.219248    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09870d81_7076_49f2_a276_6530e16cf472.slice/crio-6e5750aef680e45f5126255438adb5cd848190537d24eb61332be78c6f50c10c WatchSource:0}: Error finding container 6e5750aef680e45f5126255438adb5cd848190537d24eb61332be78c6f50c10c: Status 404 returned error can't find the container with id 6e5750aef680e45f5126255438adb5cd848190537d24eb61332be78c6f50c10c
Jul 28 17:10:21.225727 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.224500    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:21.236089 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.236064    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl"
Jul 28 17:10:21.240538 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.240520    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kq57b"
Jul 28 17:10:21.243994 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.243971    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af24ccc_78ed_445a_891a_bf25f722e76a.slice/crio-8b455260cebaa63f84bc9b9aa31c9163d6fa60fe126e00019b1f3c2072d015f5 WatchSource:0}: Error finding container 8b455260cebaa63f84bc9b9aa31c9163d6fa60fe126e00019b1f3c2072d015f5: Status 404 returned error can't find the container with id 8b455260cebaa63f84bc9b9aa31c9163d6fa60fe126e00019b1f3c2072d015f5
Jul 28 17:10:21.247540 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.247519    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e89d16_f69e_458d_848a_53e2a1dfbcdd.slice/crio-55bc9d91172578536ee9d7eb51bf0167462e7360248b6752f9872cbb467091ce WatchSource:0}: Error finding container 55bc9d91172578536ee9d7eb51bf0167462e7360248b6752f9872cbb467091ce: Status 404 returned error can't find the container with id 55bc9d91172578536ee9d7eb51bf0167462e7360248b6752f9872cbb467091ce
Jul 28 17:10:21.264595 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.264577    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pscn6"
Jul 28 17:10:21.268822 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.268802    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:21.271748 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.271729    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df21e2b_1c50_4d77_8985_2aa6b8a75c2a.slice/crio-ed900e10f1165cde84aca5ee4160d69d80c0c54c9d79473f75322aa9a6f1b313 WatchSource:0}: Error finding container ed900e10f1165cde84aca5ee4160d69d80c0c54c9d79473f75322aa9a6f1b313: Status 404 returned error can't find the container with id ed900e10f1165cde84aca5ee4160d69d80c0c54c9d79473f75322aa9a6f1b313
Jul 28 17:10:21.276409 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:21.276390    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c14d132_50ec_45ad_9967_87f75f5d4730.slice/crio-4c0f7279be5d6f7d44baf5f7a822ea48352df2aee0dff2c38f13e0c7ab93fdf6 WatchSource:0}: Error finding container 4c0f7279be5d6f7d44baf5f7a822ea48352df2aee0dff2c38f13e0c7ab93fdf6: Status 404 returned error can't find the container with id 4c0f7279be5d6f7d44baf5f7a822ea48352df2aee0dff2c38f13e0c7ab93fdf6
Jul 28 17:10:21.534533 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.534433    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:21.534533 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.534527    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:21.534705 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.534590    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.534705 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.534612    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.534705 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.534652    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.534636506 +0000 UTC m=+3.086946380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.534705 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.534668    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.534660855 +0000 UTC m=+3.086970730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.635012 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.634980    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:21.635208 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.635112    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:21.635208 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.635130    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:21.635208 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.635141    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.635208 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:21.635197    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.635177779 +0000 UTC m=+3.187487658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:21.806362 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.806279    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:21.885177 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.885050    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:21 +0000 UTC" deadline="2028-04-07 22:18:14.977889748 +0000 UTC"
Jul 28 17:10:21.885177 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.885115    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14861h7m53.09277943s"
Jul 28 17:10:21.974752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.974691    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl" event={"ID":"1af24ccc-78ed-445a-891a-bf25f722e76a","Type":"ContainerStarted","Data":"8b455260cebaa63f84bc9b9aa31c9163d6fa60fe126e00019b1f3c2072d015f5"}
Jul 28 17:10:21.977327 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.977293    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzck2" event={"ID":"069a7e39-3dc9-498d-80c2-b0ef2e353733","Type":"ContainerStarted","Data":"f2bce9803c75867516b150e8516df72ecd78144c4e4e1f41a1aa3fdf8e9cbc8e"}
Jul 28 17:10:21.980403 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.980373    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" event={"ID":"eb9f2dabc625c037e2ff47a5be458f87","Type":"ContainerStarted","Data":"3948cc4a934f21005c9c909cf1ed9d0c7ab8168748a123720b4693b8e9a27602"}
Jul 28 17:10:21.984725 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.984699    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"04b8fff3b385cbb84489843f1a6f24b4c339ca2fd95a197d22527cf638379dce"}
Jul 28 17:10:21.986900 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.986874    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pscn6" event={"ID":"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a","Type":"ContainerStarted","Data":"ed900e10f1165cde84aca5ee4160d69d80c0c54c9d79473f75322aa9a6f1b313"}
Jul 28 17:10:21.989608 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.989585    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerStarted","Data":"4a79a6db2f35d998100742bf45253c50623cb878ddddc4108db4a154fa96904d"}
Jul 28 17:10:21.993194 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.993167    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dx6mc" event={"ID":"09870d81-7076-49f2-a276-6530e16cf472","Type":"ContainerStarted","Data":"6e5750aef680e45f5126255438adb5cd848190537d24eb61332be78c6f50c10c"}
Jul 28 17:10:21.995869 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.995839    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"e1461c37f00540e8e6e35dbf0066c0e3","Type":"ContainerStarted","Data":"817daf4d2847cccaf3b0b959107edd403a700f6bdb1ea1a27687de28e24919cb"}
Jul 28 17:10:21.998668 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:21.998642    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9kq74" event={"ID":"6c14d132-50ec-45ad-9967-87f75f5d4730","Type":"ContainerStarted","Data":"4c0f7279be5d6f7d44baf5f7a822ea48352df2aee0dff2c38f13e0c7ab93fdf6"}
Jul 28 17:10:22.003220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.003198    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kq57b" event={"ID":"15e89d16-f69e-458d-848a-53e2a1dfbcdd","Type":"ContainerStarted","Data":"55bc9d91172578536ee9d7eb51bf0167462e7360248b6752f9872cbb467091ce"}
Jul 28 17:10:22.109761 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.109688    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:22.542666 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.542571    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:22.542862 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.542688    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:22.542862 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.542732    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:22.542862 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.542814    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.542792554 +0000 UTC m=+5.095102450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:22.542862 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.542835    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:22.543254 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.542906    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.542886526 +0000 UTC m=+5.095196398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:22.643815 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.643760    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:22.644014 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.643949    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:22.644014 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.643972    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:22.644014 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.643985    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:22.644212 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.644061    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:24.644023561 +0000 UTC m=+5.196333455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:22.886387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.886255    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:21 +0000 UTC" deadline="2028-01-04 13:33:45.623759259 +0000 UTC"
Jul 28 17:10:22.886387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.886300    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12596h23m22.737464203s"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.966963    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.967124    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.967554    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.967662    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:22.967738    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:22.967881 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:22.967831    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:23.011760 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:23.011724    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l8bj2" event={"ID":"b44d39e4-6f0e-43ba-962f-14f740a7bbfe","Type":"ContainerStarted","Data":"e372382a739aa8f01e1861ad0ab8e35a6d55bdf2671b7eb4f19199188f4fa26b"}
Jul 28 17:10:23.303211 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:23.303121    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:24.560931 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.560903    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:24.561433 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.560954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:24.561433 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.561144    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:24.561433 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.561204    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:28.56118549 +0000 UTC m=+9.113495375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:24.561617 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.561570    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:24.561673 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.561637    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:28.561619513 +0000 UTC m=+9.113929391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:24.662058 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.661543    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:24.662058 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.661702    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:24.662058 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.661724    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:24.662058 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.661751    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:24.662058 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.661811    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:28.661790655 +0000 UTC m=+9.214100540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:24.967411 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.967325    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:24.967411 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.967375    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:24.967626 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:24.967345    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:24.975108 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.967921    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:24.975108 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.970282    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:24.975108 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:24.970453    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:26.967073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:26.966964    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:26.967073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:26.966986    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:26.967073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:26.967020    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:26.967611 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:26.967125    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:26.967611 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:26.967533    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:26.967724 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:26.967612    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:28.594943 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.594896    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:28.595315 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.594949    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:28.595315 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.595105    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:28.595315 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.595166    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.595147477 +0000 UTC m=+17.147457351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:28.595634 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.595549    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:28.595634 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.595608    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.595591978 +0000 UTC m=+17.147901854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:28.696408 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.696245    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:28.696577 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.696442    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:28.696577 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.696464    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:28.696577 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.696477    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:28.696577 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.696538    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:36.696521793 +0000 UTC m=+17.248831662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:28.966725 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.966652    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:28.966888 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.966777    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:28.966888 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.966829    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:28.967019 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.966948    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:28.967019 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:28.966966    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:28.967129 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:28.967081    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:30.967075 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:30.967038    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:30.967515 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:30.967089    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:30.967515 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:30.967172    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:30.967515 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:30.967187    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:30.967515 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:30.967269    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:30.967515 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:30.967370    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:32.967267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:32.967233    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:32.967724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:32.967251    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:32.967724 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:32.967366    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:32.967724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:32.967413    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:32.967724 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:32.967490    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:32.967724 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:32.967583    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:34.966888 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:34.966861    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:34.967285 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:34.966969    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:34.967285 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:34.966985    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:34.967285 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:34.967094    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:34.967285 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:34.967127    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:34.967285 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:34.967183    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:36.659267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.659232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:36.659267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.659273    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:36.659722 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.659394    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:36.659722 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.659409    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.659722 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.659449    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:52.659434634 +0000 UTC m=+33.211744522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:36.659722 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.659478    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:52.659458436 +0000 UTC m=+33.211768322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:36.760451 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.760404    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:36.760627 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.760592    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:36.760627 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.760617    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:36.760627 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.760627    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.760730 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.760681    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:10:52.760664835 +0000 UTC m=+33.312974704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:36.966961 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.966880    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:36.966961 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.966925    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:36.967173 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:36.966979    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:36.967173 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.967003    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:36.967173 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.967078    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:36.967268 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:36.967186    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:38.967704 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:38.967645    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:38.967704 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:38.967705    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:38.968243 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:38.967962    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:38.968430 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:38.968412    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:38.968592 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:38.968571    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:38.968695 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:38.968678    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:40.967237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:40.967210    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:40.967570 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:40.967210    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:40.967570 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:40.967309    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:40.967570 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:40.967210    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:40.967570 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:40.967392    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:40.967570 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:40.967457    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:42.065733 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.065520    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl" event={"ID":"1af24ccc-78ed-445a-891a-bf25f722e76a","Type":"ContainerStarted","Data":"b60f4f28ae7ee78c3d7fbcbf94fdff26f87e63e171a71e5258377d11a8033688"}
Jul 28 17:10:42.066912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.066893    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rzck2" event={"ID":"069a7e39-3dc9-498d-80c2-b0ef2e353733","Type":"ContainerStarted","Data":"f76caa99bdb9b11087141bcec638105fc20a84031c432a0c88b70cccf4556d01"}
Jul 28 17:10:42.068277 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.068254    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" event={"ID":"eb9f2dabc625c037e2ff47a5be458f87","Type":"ContainerStarted","Data":"42872b890fefc8547c95a9a1ad38a38e3b348a7a47d58a46089604f8dc5cf2ed"}
Jul 28 17:10:42.070716 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.070696    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"117a4010ba202ae398f2e833fb2668e12c711ccc37d34e5fbdbc1ca936418864"}
Jul 28 17:10:42.070794 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.070754    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"fc876b2a6502c857487d2d8f187c09df1506a270634ee5d4ae64cf854d9dd927"}
Jul 28 17:10:42.070794 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.070792    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"1c7cab26139958d48127dd09f40b15fc2815313ca7a6b5186e10d11bf5a7e72e"}
Jul 28 17:10:42.070892 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.070835    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"1f3577f98399c8fc10af531c05ac08183356d17aa53e2c36f383f44bbc8712c9"}
Jul 28 17:10:42.070892 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.070873    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"e490a510944ef74d039997f3001a342793d8b2cff81e8924f62cfd63dc9ecf39"}
Jul 28 17:10:42.072186 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.072120    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pscn6" event={"ID":"1df21e2b-1c50-4d77-8985-2aa6b8a75c2a","Type":"ContainerStarted","Data":"8cd1acde98e66e7155461f6e6e70ef4f8d30d9135e108cc2d4ca1afdaf681fb5"}
Jul 28 17:10:42.073676 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.073649    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="9779ad0d0ae5b7b8b2d41d82137c86da70991df201d16c54172d27792d8840e5" exitCode=0
Jul 28 17:10:42.073759 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.073720    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"9779ad0d0ae5b7b8b2d41d82137c86da70991df201d16c54172d27792d8840e5"}
Jul 28 17:10:42.075124 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.075102    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dx6mc" event={"ID":"09870d81-7076-49f2-a276-6530e16cf472","Type":"ContainerStarted","Data":"353b7429029288eb3c2b21d86b1d0c7ad2167edcfecd4f824fd34ca4efbbe5a0"}
Jul 28 17:10:42.076618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.076595    2552 generic.go:356] "Generic (PLEG): container finished" podID="e1461c37f00540e8e6e35dbf0066c0e3" containerID="7d6a6bb7197aef4ed9e521c889e1f38d5a6559579ea4a17dbd4db7fc3ed7311e" exitCode=0
Jul 28 17:10:42.076754 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.076680    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"e1461c37f00540e8e6e35dbf0066c0e3","Type":"ContainerDied","Data":"7d6a6bb7197aef4ed9e521c889e1f38d5a6559579ea4a17dbd4db7fc3ed7311e"}
Jul 28 17:10:42.078226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.078205    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l8bj2" event={"ID":"b44d39e4-6f0e-43ba-962f-14f740a7bbfe","Type":"ContainerStarted","Data":"37d06d1f8368c12b6ce849daeb16b2f2bab40d9d51941da1ede99f8b164d413d"}
Jul 28 17:10:42.079565 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.079495    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9kq74" event={"ID":"6c14d132-50ec-45ad-9967-87f75f5d4730","Type":"ContainerStarted","Data":"7997cdbff726f622a9f6ede241b5b782548724bec371443ddf028946488be968"}
Jul 28 17:10:42.091912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.091875    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rzck2" podStartSLOduration=2.159524812 podStartE2EDuration="22.091863944s" totalImagesPullingTime="19.932339132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.091392795 +0000 UTC m=+22.643702685" watchObservedRunningTime="2026-07-28 17:10:42.091863944 +0000 UTC m=+22.644173835"
Jul 28 17:10:42.126023 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.125962    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-80.ec2.internal" podStartSLOduration=21.125944297 podStartE2EDuration="21.125944297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.125560766 +0000 UTC m=+22.677870658" watchObservedRunningTime="2026-07-28 17:10:42.125944297 +0000 UTC m=+22.678254188"
Jul 28 17:10:42.169417 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.169376    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l8bj2" podStartSLOduration=3.16929895 podStartE2EDuration="22.169361609s" totalImagesPullingTime="19.000062659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.147661939 +0000 UTC m=+22.699971832" watchObservedRunningTime="2026-07-28 17:10:42.169361609 +0000 UTC m=+22.721671499"
Jul 28 17:10:42.192692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.192642    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pscn6" podStartSLOduration=2.309664626 podStartE2EDuration="22.192626818s" totalImagesPullingTime="19.882962192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.169847229 +0000 UTC m=+22.722157119" watchObservedRunningTime="2026-07-28 17:10:42.192626818 +0000 UTC m=+22.744936709"
Jul 28 17:10:42.230132 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.229919    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dx6mc" podStartSLOduration=2.2975610189999998 podStartE2EDuration="22.2299039s" totalImagesPullingTime="19.932342881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.193079115 +0000 UTC m=+22.745389005" watchObservedRunningTime="2026-07-28 17:10:42.2299039 +0000 UTC m=+22.782213782"
Jul 28 17:10:42.967115 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.967040    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:42.967279 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.967116    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:42.967279 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:42.967116    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:42.967279 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:42.967217    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:42.967430 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:42.967339    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:42.967430 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:42.967415    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:43.085585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.085542    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"c7210984b61069153973fa8970a1f79374f365a6bf7205d52c5d6196d921b554"}
Jul 28 17:10:43.087556 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.087529    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" event={"ID":"e1461c37f00540e8e6e35dbf0066c0e3","Type":"ContainerStarted","Data":"32ac25a22f982e663894d56e4a5811affdf8b77f599bf6e5d577a6b99082ad27"}
Jul 28 17:10:43.089227 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.089196    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kq57b" event={"ID":"15e89d16-f69e-458d-848a-53e2a1dfbcdd","Type":"ContainerStarted","Data":"b7803551da4eadc73cfb18edfac8f7e9cb299cc109161fbc2935ca2aa8b7d49b"}
Jul 28 17:10:43.106676 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.106639    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-80.ec2.internal" podStartSLOduration=22.106626259 podStartE2EDuration="22.106626259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:43.106231598 +0000 UTC m=+23.658541490" watchObservedRunningTime="2026-07-28 17:10:43.106626259 +0000 UTC m=+23.658936149"
Jul 28 17:10:43.106881 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.106854    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9kq74" podStartSLOduration=3.256736469 podStartE2EDuration="23.106845457s" totalImagesPullingTime="19.850108988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:42.248943911 +0000 UTC m=+22.801253802" watchObservedRunningTime="2026-07-28 17:10:43.106845457 +0000 UTC m=+23.659155350"
Jul 28 17:10:43.115596 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.115570    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:43.116103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.116085    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:43.127688 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.127641    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kq57b" podStartSLOduration=3.248736799 podStartE2EDuration="23.12762669s" totalImagesPullingTime="19.878889891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:43.127210047 +0000 UTC m=+23.679519939" watchObservedRunningTime="2026-07-28 17:10:43.12762669 +0000 UTC m=+23.679936580"
Jul 28 17:10:43.232087 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:43.232050    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:10:44.013159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.013047    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:10:43.232075385Z","UUID":"f9abe04e-c566-4107-8bce-c2c96efa0615","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:10:44.014739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.014649    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:10:44.014739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.014695    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:10:44.095092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.095050    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl" event={"ID":"1af24ccc-78ed-445a-891a-bf25f722e76a","Type":"ContainerStarted","Data":"acc8f0608ccf651f894491fd4cdc4e3b129d32b96c9b383e32510103efd5908e"}
Jul 28 17:10:44.096132 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.095350    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:44.096132 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.095744    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9kq74"
Jul 28 17:10:44.967501 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.967448    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:44.967501 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.967500    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:44.967731 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:44.967451    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:44.967731 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:44.967563    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:44.967731 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:44.967644    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:44.967731 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:44.967718    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:45.100848 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:45.100796    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"93552e4458bfd86108ad565c650f3bc7739aeb8fda0bf7b44ea06ae8f6fbc1de"}
Jul 28 17:10:45.103136 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:45.103110    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl" event={"ID":"1af24ccc-78ed-445a-891a-bf25f722e76a","Type":"ContainerStarted","Data":"bf2f4757890181d8804234b32688aac4d02bf4868d177767f04daf7cefa5a0d0"}
Jul 28 17:10:46.967655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:46.967497    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:46.968215 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:46.967497    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:46.968215 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:46.967730    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:46.968215 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:46.967497    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:46.968215 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:46.967784    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:46.968215 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:46.967846    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:47.111028 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.110975    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" event={"ID":"8bf73946-55d9-4630-8bc2-f6c27634a26c","Type":"ContainerStarted","Data":"41e5205f605d508e5cef1d62840accec12652620ef760e29e3f7f457b5d8c7c9"}
Jul 28 17:10:47.111244 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.111229    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:47.111324 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.111310    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:47.112850 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.112824    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="dab5123f88f36095052faa5cfeec8f0f55a739c91b3cf71db050977c61ea4477" exitCode=0
Jul 28 17:10:47.112952 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.112868    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"dab5123f88f36095052faa5cfeec8f0f55a739c91b3cf71db050977c61ea4477"}
Jul 28 17:10:47.127124 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.127106    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:47.145520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.145485    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7" podStartSLOduration=7.631956259 podStartE2EDuration="27.145473383s" totalImagesPullingTime="19.513517124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:47.143824837 +0000 UTC m=+27.696134728" watchObservedRunningTime="2026-07-28 17:10:47.145473383 +0000 UTC m=+27.697783265"
Jul 28 17:10:47.146065 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:47.146022    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-slnnl" podStartSLOduration=4.9156199229999995 podStartE2EDuration="27.146009363s" totalImagesPullingTime="22.23038944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.131934087 +0000 UTC m=+25.684243979" watchObservedRunningTime="2026-07-28 17:10:47.146009363 +0000 UTC m=+27.698319247"
Jul 28 17:10:48.117969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.117930    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:48.132472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.132445    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:10:48.480213 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.479809    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4wzj"]
Jul 28 17:10:48.480466 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.480306    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:48.480466 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:48.480407    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:48.480654 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.480490    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dww89"]
Jul 28 17:10:48.480654 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.480573    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:48.480742 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:48.480659    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:48.481366 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.481343    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pdtks"]
Jul 28 17:10:48.481491 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:48.481475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:48.481593 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:48.481572    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:49.121224 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:49.121191    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="9930d3a38ed84940f4dccb0f0761b2fc6182f3a732efb19957338c57995ed049" exitCode=0
Jul 28 17:10:49.121613 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:49.121272    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"9930d3a38ed84940f4dccb0f0761b2fc6182f3a732efb19957338c57995ed049"}
Jul 28 17:10:49.967882 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:49.967846    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:49.968055 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:49.967945    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:49.968055 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:49.967980    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:49.968055 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:49.968013    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:49.968183 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:49.968087    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:49.968183 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:49.968158    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:51.127527 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:51.127487    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="fa0b1d3bf0fec96922a037526d638608fd3acb3ce59df55f93ce75e59e0618e2" exitCode=0
Jul 28 17:10:51.127893 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:51.127548    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"fa0b1d3bf0fec96922a037526d638608fd3acb3ce59df55f93ce75e59e0618e2"}
Jul 28 17:10:51.967267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:51.967234    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:51.967267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:51.967261    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:51.967267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:51.967234    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:51.967560 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:51.967381    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:51.967560 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:51.967443    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:51.967560 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:51.967517    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:52.679840 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:52.679786    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:52.680322 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:52.679916    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:52.680322 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.679919    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:52.680322 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.679984    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs podName:52ed84b2-9963-498c-8594-54462ee49011 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.679960294 +0000 UTC m=+65.232270171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs") pod "network-metrics-daemon-dww89" (UID: "52ed84b2-9963-498c-8594-54462ee49011") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:52.680322 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.680061    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:52.680322 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.680111    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret podName:51709ba2-21ed-49d5-8a4f-f4c2b1f92745 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.680098973 +0000 UTC m=+65.232408841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret") pod "global-pull-secret-syncer-pdtks" (UID: "51709ba2-21ed-49d5-8a4f-f4c2b1f92745") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:52.781190 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:52.781154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:52.781349 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.781297    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:52.781349 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.781317    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:52.781349 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.781328    2552 projected.go:196] Error preparing data for projected volume kube-api-access-9k9vs for pod openshift-network-diagnostics/network-check-target-g4wzj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:52.781466 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:52.781381    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs podName:3d1de778-9e38-4e7d-927e-6d8b4d89878a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.781367668 +0000 UTC m=+65.333677561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9k9vs" (UniqueName: "kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs") pod "network-check-target-g4wzj" (UID: "3d1de778-9e38-4e7d-927e-6d8b4d89878a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:53.967196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:53.967164    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:53.967196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:53.967186    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:53.967717 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:53.967166    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:53.967717 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:53.967298    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dww89" podUID="52ed84b2-9963-498c-8594-54462ee49011"
Jul 28 17:10:53.967717 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:53.967371    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdtks" podUID="51709ba2-21ed-49d5-8a4f-f4c2b1f92745"
Jul 28 17:10:53.967717 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:53.967442    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g4wzj" podUID="3d1de778-9e38-4e7d-927e-6d8b4d89878a"
Jul 28 17:10:55.288189 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.288022    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-80.ec2.internal" event="NodeReady"
Jul 28 17:10:55.288602 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.288287    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:10:55.350126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.350100    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7nspz"]
Jul 28 17:10:55.365448 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.365422    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cm88p"]
Jul 28 17:10:55.365722 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.365598    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.369285 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.369263    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hqvdh\""
Jul 28 17:10:55.369559 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.369537    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:10:55.369782 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.369760    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:10:55.389672 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.389655    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nspz"]
Jul 28 17:10:55.389758 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.389686    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cm88p"]
Jul 28 17:10:55.389802 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.389764    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.392927 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.392912    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:10:55.393141 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.393126    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:10:55.393553 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.393536    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:10:55.394007 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.393968    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cpt5q\""
Jul 28 17:10:55.501968 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.501938    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.501968 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.501970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed188f2a-8cb8-47dd-9389-e46104e0de0b-cert\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.502183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.502062    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2019a327-9b47-402f-9579-4015f8476743-tmp-dir\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.502183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.502090    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6k4\" (UniqueName: \"kubernetes.io/projected/ed188f2a-8cb8-47dd-9389-e46104e0de0b-kube-api-access-pf6k4\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.502183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.502117    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2019a327-9b47-402f-9579-4015f8476743-config-volume\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.502183 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.502158    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lf5\" (UniqueName: \"kubernetes.io/projected/2019a327-9b47-402f-9579-4015f8476743-kube-api-access-z4lf5\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603258 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6k4\" (UniqueName: \"kubernetes.io/projected/ed188f2a-8cb8-47dd-9389-e46104e0de0b-kube-api-access-pf6k4\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.603385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603265    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2019a327-9b47-402f-9579-4015f8476743-config-volume\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lf5\" (UniqueName: \"kubernetes.io/projected/2019a327-9b47-402f-9579-4015f8476743-kube-api-access-z4lf5\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603349    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed188f2a-8cb8-47dd-9389-e46104e0de0b-cert\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.603612 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:55.603466    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:55.603612 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:55.603530    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls podName:2019a327-9b47-402f-9579-4015f8476743 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:56.103509838 +0000 UTC m=+36.655819712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls") pod "dns-default-7nspz" (UID: "2019a327-9b47-402f-9579-4015f8476743") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:55.603612 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603579    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2019a327-9b47-402f-9579-4015f8476743-tmp-dir\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603795    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2019a327-9b47-402f-9579-4015f8476743-tmp-dir\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.603922 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.603902    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2019a327-9b47-402f-9579-4015f8476743-config-volume\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.607499 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.607474    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed188f2a-8cb8-47dd-9389-e46104e0de0b-cert\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.615897 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.615873    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lf5\" (UniqueName: \"kubernetes.io/projected/2019a327-9b47-402f-9579-4015f8476743-kube-api-access-z4lf5\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:55.615977 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.615892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6k4\" (UniqueName: \"kubernetes.io/projected/ed188f2a-8cb8-47dd-9389-e46104e0de0b-kube-api-access-pf6k4\") pod \"ingress-canary-cm88p\" (UID: \"ed188f2a-8cb8-47dd-9389-e46104e0de0b\") " pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.697768 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.697736    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cm88p"
Jul 28 17:10:55.904292 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.904250    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:55.913364 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.913342    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cm88p"]
Jul 28 17:10:55.917375 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:10:55.917351    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded188f2a_8cb8_47dd_9389_e46104e0de0b.slice/crio-c000d9b07d8591b56f65ee618f379dc3817573c8566f6a12245d593162c0adcd WatchSource:0}: Error finding container c000d9b07d8591b56f65ee618f379dc3817573c8566f6a12245d593162c0adcd: Status 404 returned error can't find the container with id c000d9b07d8591b56f65ee618f379dc3817573c8566f6a12245d593162c0adcd
Jul 28 17:10:55.967103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.967073    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:10:55.967220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.967077    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:10:55.967281 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.967081    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:10:55.970265 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.970248    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:55.970356 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.970323    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wz9mb\""
Jul 28 17:10:55.970427 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.970377    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:10:55.970427 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.970378    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:10:55.971585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.971569    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:55.971634 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.971575    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xf5sm\""
Jul 28 17:10:55.994645 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:55.994621    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:56.107700 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.107636    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:56.107809 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:56.107760    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:56.107847 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:56.107823    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls podName:2019a327-9b47-402f-9579-4015f8476743 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:57.10780494 +0000 UTC m=+37.660114823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls") pod "dns-default-7nspz" (UID: "2019a327-9b47-402f-9579-4015f8476743") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:56.139172 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.139143    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cm88p" event={"ID":"ed188f2a-8cb8-47dd-9389-e46104e0de0b","Type":"ContainerStarted","Data":"c000d9b07d8591b56f65ee618f379dc3817573c8566f6a12245d593162c0adcd"}
Jul 28 17:10:56.141708 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.141685    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="87abd51d795e3c45b6e9906efb6f1026401688b887671f61f5fd6c91f9a0f693" exitCode=0
Jul 28 17:10:56.141795 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.141726    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"87abd51d795e3c45b6e9906efb6f1026401688b887671f61f5fd6c91f9a0f693"}
Jul 28 17:10:56.268654 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.268628    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:56.370785 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.370699    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:56.432698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:56.432673    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:57.115171 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.115142    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:57.115385 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:57.115305    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:57.115457 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:57.115383    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls podName:2019a327-9b47-402f-9579-4015f8476743 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:59.115359858 +0000 UTC m=+39.667669729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls") pod "dns-default-7nspz" (UID: "2019a327-9b47-402f-9579-4015f8476743") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:57.147830 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.147797    2552 generic.go:356] "Generic (PLEG): container finished" podID="4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963" containerID="cc46a4fff35149427ec3ac58ee0b4c2d463290417419eb7c57518af20dac4e26" exitCode=0
Jul 28 17:10:57.147985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.147850    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerDied","Data":"cc46a4fff35149427ec3ac58ee0b4c2d463290417419eb7c57518af20dac4e26"}
Jul 28 17:10:57.317774 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.317742    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:57.415055 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.414936    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:57.470675 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:57.470639    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.156605 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.156568    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zblqh" event={"ID":"4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963","Type":"ContainerStarted","Data":"408d3984344fc72bf30f5ef681e5c83de19d2f14386155534576ecf5481d5159"}
Jul 28 17:10:58.183719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.183662    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zblqh" podStartSLOduration=7.348005948 podStartE2EDuration="38.183644856s" totalImagesPullingTime="30.835638908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:58.182118469 +0000 UTC m=+38.734428359" watchObservedRunningTime="2026-07-28 17:10:58.183644856 +0000 UTC m=+38.735954749"
Jul 28 17:10:58.305735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.305658    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.367705 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.367676    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.584375 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.584304    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.683325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.683290    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.784026 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.783972    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.887932 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.887848    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:58.985550 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:58.985522    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.132512 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.132485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:10:59.132661 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:59.132610    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:59.132701 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:10:59.132675    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls podName:2019a327-9b47-402f-9579-4015f8476743 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:03.132657527 +0000 UTC m=+43.684967606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls") pod "dns-default-7nspz" (UID: "2019a327-9b47-402f-9579-4015f8476743") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:59.143674 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.143605    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.160324 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.160300    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cm88p" event={"ID":"ed188f2a-8cb8-47dd-9389-e46104e0de0b","Type":"ContainerStarted","Data":"4a288a0841afd4596dd3098c17aaad18bc47aa9c886aa203d44ffda3d0ad2bbb"}
Jul 28 17:10:59.178479 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.178444    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cm88p" podStartSLOduration=1.491942616 podStartE2EDuration="4.178430195s" totalImagesPullingTime="2.686487579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:59.17792937 +0000 UTC m=+39.730239261" watchObservedRunningTime="2026-07-28 17:10:59.178430195 +0000 UTC m=+39.730740064"
Jul 28 17:10:59.244257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.244231    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.358251 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.358214    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.448366 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.448265    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.595625 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.595595    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"]
Jul 28 17:10:59.621857 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.621827    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"]
Jul 28 17:10:59.621985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.621919    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"
Jul 28 17:10:59.624619 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.624600    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-68bzt\""
Jul 28 17:10:59.624729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.624600    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:59.625842 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.625825    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:10:59.664817 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.664787    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.738036 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.738011    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnf7\" (UniqueName: \"kubernetes.io/projected/f81d3091-4ca9-47a1-a738-2ba312ef1ed6-kube-api-access-jvnf7\") pod \"migrator-6865c9967-gj2pm\" (UID: \"f81d3091-4ca9-47a1-a738-2ba312ef1ed6\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"
Jul 28 17:10:59.741525 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.741502    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:59.838589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.838563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnf7\" (UniqueName: \"kubernetes.io/projected/f81d3091-4ca9-47a1-a738-2ba312ef1ed6-kube-api-access-jvnf7\") pod \"migrator-6865c9967-gj2pm\" (UID: \"f81d3091-4ca9-47a1-a738-2ba312ef1ed6\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"
Jul 28 17:10:59.848871 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.848849    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnf7\" (UniqueName: \"kubernetes.io/projected/f81d3091-4ca9-47a1-a738-2ba312ef1ed6-kube-api-access-jvnf7\") pod \"migrator-6865c9967-gj2pm\" (UID: \"f81d3091-4ca9-47a1-a738-2ba312ef1ed6\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"
Jul 28 17:10:59.930354 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:10:59.930328    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"
Jul 28 17:11:00.066161 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:00.066115    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm"]
Jul 28 17:11:00.069870 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:00.069846    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81d3091_4ca9_47a1_a738_2ba312ef1ed6.slice/crio-bf351830c96d19ef7d4c393b7d8688f09c2844971ad06b046a37dc01d272a469 WatchSource:0}: Error finding container bf351830c96d19ef7d4c393b7d8688f09c2844971ad06b046a37dc01d272a469: Status 404 returned error can't find the container with id bf351830c96d19ef7d4c393b7d8688f09c2844971ad06b046a37dc01d272a469
Jul 28 17:11:00.163149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:00.163116    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm" event={"ID":"f81d3091-4ca9-47a1-a738-2ba312ef1ed6","Type":"ContainerStarted","Data":"bf351830c96d19ef7d4c393b7d8688f09c2844971ad06b046a37dc01d272a469"}
Jul 28 17:11:01.045552 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:01.045519    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:01.145566 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:01.145529    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:02.019369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:02.019346    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:02.169535 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:02.169456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm" event={"ID":"f81d3091-4ca9-47a1-a738-2ba312ef1ed6","Type":"ContainerStarted","Data":"04d63e0711c63367d22a81611e1f5ba2e6b33d28882c5c5a24769b5ccfeedf03"}
Jul 28 17:11:02.169535 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:02.169521    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm" event={"ID":"f81d3091-4ca9-47a1-a738-2ba312ef1ed6","Type":"ContainerStarted","Data":"f7167743b95078e88254df312b77fcb9706766861724d1c3cd738241520a7901"}
Jul 28 17:11:02.191580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:02.191535    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gj2pm" podStartSLOduration=1.4899748879999999 podStartE2EDuration="3.191520843s" totalImagesPullingTime="1.701545955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:02.188966755 +0000 UTC m=+42.741276657" watchObservedRunningTime="2026-07-28 17:11:02.191520843 +0000 UTC m=+42.743830733"
Jul 28 17:11:03.160682 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:03.160650    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:03.160833 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:03.160806    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:03.160887 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:03.160867    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls podName:2019a327-9b47-402f-9579-4015f8476743 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:11.160853086 +0000 UTC m=+51.713162956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls") pod "dns-default-7nspz" (UID: "2019a327-9b47-402f-9579-4015f8476743") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:05.083734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.083700    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.262912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.262881    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.363214 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.363136    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.479518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.479479    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.567983 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.567953    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.731588 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.731558    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:05.833521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:05.833485    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:06.781900 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:06.781836    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.179049 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:07.178955    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.378136 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:07.378099    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.583090 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:07.583055    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:07.811763 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:07.811733    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.696080 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:08.696044    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.795603 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:08.795572    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:08.977778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:08.977748    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.175574 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:09.175529    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.371941 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:09.371848    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.575108 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:09.575076    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.698771 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:09.698693    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:09.829816 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:09.829785    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:10.777868 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:10.777831    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.183824 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.183735    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.216259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.216227    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:11.218527 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.218505    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2019a327-9b47-402f-9579-4015f8476743-metrics-tls\") pod \"dns-default-7nspz\" (UID: \"2019a327-9b47-402f-9579-4015f8476743\") " pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:11.276837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.276784    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:11.371138 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.371108    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.398795 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.398770    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nspz"]
Jul 28 17:11:11.401691 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:11.401668    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2019a327_9b47_402f_9579_4015f8476743.slice/crio-c79eff2bc3e0db41ffa14f200f09943b143332ef08faa93efc9f0d5e94faccbe WatchSource:0}: Error finding container c79eff2bc3e0db41ffa14f200f09943b143332ef08faa93efc9f0d5e94faccbe: Status 404 returned error can't find the container with id c79eff2bc3e0db41ffa14f200f09943b143332ef08faa93efc9f0d5e94faccbe
Jul 28 17:11:11.472094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.472066    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.574848 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.574818    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.975916 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:11.975884    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.193021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:12.192974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nspz" event={"ID":"2019a327-9b47-402f-9579-4015f8476743","Type":"ContainerStarted","Data":"c79eff2bc3e0db41ffa14f200f09943b143332ef08faa93efc9f0d5e94faccbe"}
Jul 28 17:11:12.781296 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:12.781260    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.179917 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:13.179830    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.379948 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:13.379912    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.575649 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:13.575620    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.680102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:13.680060    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:13.781859 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:13.781831    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.200262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.200163    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nspz" event={"ID":"2019a327-9b47-402f-9579-4015f8476743","Type":"ContainerStarted","Data":"78dd5d311e79ef3ce00fa3fb10eec6ccacce73cf5ca2f1a2e6aa06d54443c9e4"}
Jul 28 17:11:14.200262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.200225    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nspz" event={"ID":"2019a327-9b47-402f-9579-4015f8476743","Type":"ContainerStarted","Data":"686418a782cdaaf82121caa7444d0633d6aedb633f561392c4e021b7729cc416"}
Jul 28 17:11:14.200262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.200261    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:14.221739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.221699    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7nspz" podStartSLOduration=17.202508305 podStartE2EDuration="19.221685952s" totalImagesPullingTime="2.019177647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:14.21994621 +0000 UTC m=+54.772256114" watchObservedRunningTime="2026-07-28 17:11:14.221685952 +0000 UTC m=+54.773995821"
Jul 28 17:11:14.778808 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.778778    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:14.983244 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:14.983211    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.170802 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:15.170717    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:15.372781 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:15.372750    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:20.133772 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:20.133747    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6lk7"
Jul 28 17:11:21.457601 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.457566    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"]
Jul 28 17:11:21.460159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.460142    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"]
Jul 28 17:11:21.460282 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.460265    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.461673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.461653    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.462980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.462959    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w6dfs\""
Jul 28 17:11:21.463111 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.463022    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:21.464271 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.464251    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:21.464385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.464279    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 17:11:21.464527 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.464513    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:11:21.464673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.464657    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-j6pv8\""
Jul 28 17:11:21.464861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.464842    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:11:21.465403 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.465388    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:11:21.465403 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.465397    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 17:11:21.480072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.480054    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"]
Jul 28 17:11:21.483752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.483734    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"]
Jul 28 17:11:21.485441 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.485422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk82z\" (UniqueName: \"kubernetes.io/projected/9572603a-e834-423e-999e-a28e5d3f5ce5-kube-api-access-nk82z\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.485492 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.485478    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bcb2c83f-6bab-486f-a5d5-e4cba341d168-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.485526 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.485501    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4slwr\" (UniqueName: \"kubernetes.io/projected/bcb2c83f-6bab-486f-a5d5-e4cba341d168-kube-api-access-4slwr\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.485526 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.485519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9572603a-e834-423e-999e-a28e5d3f5ce5-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.563713 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.563682    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"]
Jul 28 17:11:21.565459 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.565441    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.569549 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.569534    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 17:11:21.586432 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586407    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9572603a-e834-423e-999e-a28e5d3f5ce5-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.586532 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586457    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk82z\" (UniqueName: \"kubernetes.io/projected/9572603a-e834-423e-999e-a28e5d3f5ce5-kube-api-access-nk82z\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.586532 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586504    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bf7f0-d231-42de-b824-9a4c756658f6-tmp\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.586624 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586528    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfg5\" (UniqueName: \"kubernetes.io/projected/882bf7f0-d231-42de-b824-9a4c756658f6-kube-api-access-hmfg5\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.586704 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586686    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bcb2c83f-6bab-486f-a5d5-e4cba341d168-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.586746 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586720    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/882bf7f0-d231-42de-b824-9a4c756658f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.586800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.586746    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4slwr\" (UniqueName: \"kubernetes.io/projected/bcb2c83f-6bab-486f-a5d5-e4cba341d168-kube-api-access-4slwr\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.588738 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.588718    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9572603a-e834-423e-999e-a28e5d3f5ce5-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.588926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.588907    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bcb2c83f-6bab-486f-a5d5-e4cba341d168-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.613730 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.613710    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"]
Jul 28 17:11:21.614688 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.614665    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4slwr\" (UniqueName: \"kubernetes.io/projected/bcb2c83f-6bab-486f-a5d5-e4cba341d168-kube-api-access-4slwr\") pod \"managed-serviceaccount-addon-agent-5c5d44656b-fc7sc\" (UID: \"bcb2c83f-6bab-486f-a5d5-e4cba341d168\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.637002 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.636968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk82z\" (UniqueName: \"kubernetes.io/projected/9572603a-e834-423e-999e-a28e5d3f5ce5-kube-api-access-nk82z\") pod \"cluster-samples-operator-6748489656-5wz64\" (UID: \"9572603a-e834-423e-999e-a28e5d3f5ce5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.686955 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.686932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bf7f0-d231-42de-b824-9a4c756658f6-tmp\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.687068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.686960    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfg5\" (UniqueName: \"kubernetes.io/projected/882bf7f0-d231-42de-b824-9a4c756658f6-kube-api-access-hmfg5\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.687068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.687022    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/882bf7f0-d231-42de-b824-9a4c756658f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.687342 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.687326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/882bf7f0-d231-42de-b824-9a4c756658f6-tmp\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.689500 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.689481    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/882bf7f0-d231-42de-b824-9a4c756658f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.718828 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.718776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfg5\" (UniqueName: \"kubernetes.io/projected/882bf7f0-d231-42de-b824-9a4c756658f6-kube-api-access-hmfg5\") pod \"klusterlet-addon-workmgr-847c7cd69d-gjmb2\" (UID: \"882bf7f0-d231-42de-b824-9a4c756658f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.761093 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.761051    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.770907 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.770886    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"
Jul 28 17:11:21.786877 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.786855    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"
Jul 28 17:11:21.788057 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.788033    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d5779b67c-555dc"]
Jul 28 17:11:21.789877 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.789860    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.800885 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.800866    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5xqg\""
Jul 28 17:11:21.801102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.801083    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:11:21.801481 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.801469    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:11:21.801521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.801472    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:11:21.809261 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.809239    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:11:21.816835 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.816334    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d5779b67c-555dc"]
Jul 28 17:11:21.874779 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.874181    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:21.888158 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsqxk\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-kube-api-access-jsqxk\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888170    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-image-registry-private-configuration\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888199    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-tls\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888251    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-certificates\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888289    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-installation-pull-secrets\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888494 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888345    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-trusted-ca\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888494 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888361    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-bound-sa-token\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.888494 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.888390    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390ef382-926e-4dde-a1c5-874ccf6cdf93-ca-trust-extracted\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.899760 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.899721    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.937679 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.937653    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc"]
Jul 28 17:11:21.940930 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:21.940904    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcb2c83f_6bab_486f_a5d5_e4cba341d168.slice/crio-f5313144cf7d084dfe0124fa820380a8af15a8bc7bc29b6efbf08b9b8bf9a48a WatchSource:0}: Error finding container f5313144cf7d084dfe0124fa820380a8af15a8bc7bc29b6efbf08b9b8bf9a48a: Status 404 returned error can't find the container with id f5313144cf7d084dfe0124fa820380a8af15a8bc7bc29b6efbf08b9b8bf9a48a
Jul 28 17:11:21.946057 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.946033    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64"]
Jul 28 17:11:21.985600 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.985565    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:21.988781 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.988753    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-installation-pull-secrets\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.988821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-trusted-ca\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.988844    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-bound-sa-token\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.988881    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390ef382-926e-4dde-a1c5-874ccf6cdf93-ca-trust-extracted\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.989129    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsqxk\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-kube-api-access-jsqxk\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.989180    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-image-registry-private-configuration\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.989208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-tls\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989386 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.989235    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-certificates\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.989386 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.989334    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/390ef382-926e-4dde-a1c5-874ccf6cdf93-ca-trust-extracted\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.990159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.990138    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-trusted-ca\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.990370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.990352    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-certificates\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.991493 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.991472    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-installation-pull-secrets\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.991956 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.991932    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-registry-tls\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:21.992079 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:21.991935    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/390ef382-926e-4dde-a1c5-874ccf6cdf93-image-registry-private-configuration\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:22.008320 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.008299    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"]
Jul 28 17:11:22.010543 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.010249    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsqxk\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-kube-api-access-jsqxk\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:22.012281 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.012257    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/390ef382-926e-4dde-a1c5-874ccf6cdf93-bound-sa-token\") pod \"image-registry-7d5779b67c-555dc\" (UID: \"390ef382-926e-4dde-a1c5-874ccf6cdf93\") " pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:22.012588 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:22.012564    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882bf7f0_d231_42de_b824_9a4c756658f6.slice/crio-016f683c1945e4ac49fffe9ad960bce9ef5911630b2cf50355e1e80b92b950d8 WatchSource:0}: Error finding container 016f683c1945e4ac49fffe9ad960bce9ef5911630b2cf50355e1e80b92b950d8: Status 404 returned error can't find the container with id 016f683c1945e4ac49fffe9ad960bce9ef5911630b2cf50355e1e80b92b950d8
Jul 28 17:11:22.089718 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.089690    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.098668 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.098652    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:22.219146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.219111    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2" event={"ID":"882bf7f0-d231-42de-b824-9a4c756658f6","Type":"ContainerStarted","Data":"016f683c1945e4ac49fffe9ad960bce9ef5911630b2cf50355e1e80b92b950d8"}
Jul 28 17:11:22.220111 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.220080    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc" event={"ID":"bcb2c83f-6bab-486f-a5d5-e4cba341d168","Type":"ContainerStarted","Data":"f5313144cf7d084dfe0124fa820380a8af15a8bc7bc29b6efbf08b9b8bf9a48a"}
Jul 28 17:11:22.221092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.221040    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64" event={"ID":"9572603a-e834-423e-999e-a28e5d3f5ce5","Type":"ContainerStarted","Data":"0d09840b57e454bb938820808d059db792007bcf21dbba36ef6a9fc7ea725d7d"}
Jul 28 17:11:22.273972 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:22.273948    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390ef382_926e_4dde_a1c5_874ccf6cdf93.slice/crio-48131cc874c793628e543a8dc82b6910f527649cb92bda250167e036347a9b9e WatchSource:0}: Error finding container 48131cc874c793628e543a8dc82b6910f527649cb92bda250167e036347a9b9e: Status 404 returned error can't find the container with id 48131cc874c793628e543a8dc82b6910f527649cb92bda250167e036347a9b9e
Jul 28 17:11:22.277679 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.277657    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d5779b67c-555dc"]
Jul 28 17:11:22.495423 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.495389    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:22.568108 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:22.568024    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:23.229519 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:23.229468    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d5779b67c-555dc" event={"ID":"390ef382-926e-4dde-a1c5-874ccf6cdf93","Type":"ContainerStarted","Data":"7a389cb7c73fffd90bdf7bf98570b768ff96e10eb09af51efc4297f46aed3cf8"}
Jul 28 17:11:23.229695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:23.229530    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d5779b67c-555dc" event={"ID":"390ef382-926e-4dde-a1c5-874ccf6cdf93","Type":"ContainerStarted","Data":"48131cc874c793628e543a8dc82b6910f527649cb92bda250167e036347a9b9e"}
Jul 28 17:11:23.229695 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:23.229572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:23.260142 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:23.260092    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d5779b67c-555dc" podStartSLOduration=2.260078428 podStartE2EDuration="2.260078428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:23.259123528 +0000 UTC m=+63.811433437" watchObservedRunningTime="2026-07-28 17:11:23.260078428 +0000 UTC m=+63.812388319"
Jul 28 17:11:24.205047 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.205020    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7nspz"
Jul 28 17:11:24.235543 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.235504    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64" event={"ID":"9572603a-e834-423e-999e-a28e5d3f5ce5","Type":"ContainerStarted","Data":"49a718a59b07919c931f3fd4cb7c3a34dadcc6afd03205671652f687ea9d4975"}
Jul 28 17:11:24.235693 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.235571    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64" event={"ID":"9572603a-e834-423e-999e-a28e5d3f5ce5","Type":"ContainerStarted","Data":"65ac53a97e0fde6642d058d40e114a59c491d45c4ebce029f7d4e62e8c930a9d"}
Jul 28 17:11:24.260538 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.260495    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5wz64" podStartSLOduration=1.631847798 podStartE2EDuration="3.260480547s" totalImagesPullingTime="1.628632749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:24.259788974 +0000 UTC m=+64.812098859" watchObservedRunningTime="2026-07-28 17:11:24.260480547 +0000 UTC m=+64.812790475"
Jul 28 17:11:24.382142 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.382105    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.497526 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.497448    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:24.705734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.705705    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:11:24.705915 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.705749    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:11:24.708451 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.708433    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:24.709495 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.709479    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:24.718774 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.718745    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ed84b2-9963-498c-8594-54462ee49011-metrics-certs\") pod \"network-metrics-daemon-dww89\" (UID: \"52ed84b2-9963-498c-8594-54462ee49011\") " pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:11:24.718861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.718815    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/51709ba2-21ed-49d5-8a4f-f4c2b1f92745-original-pull-secret\") pod \"global-pull-secret-syncer-pdtks\" (UID: \"51709ba2-21ed-49d5-8a4f-f4c2b1f92745\") " pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:11:24.783281 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.783224    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdtks"
Jul 28 17:11:24.791038 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.791021    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xf5sm\""
Jul 28 17:11:24.798601 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.798583    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dww89"
Jul 28 17:11:24.806700 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.806674    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:11:24.809545 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.809526    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:24.820449 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.820431    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:24.833277 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.833255    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9vs\" (UniqueName: \"kubernetes.io/projected/3d1de778-9e38-4e7d-927e-6d8b4d89878a-kube-api-access-9k9vs\") pod \"network-check-target-g4wzj\" (UID: \"3d1de778-9e38-4e7d-927e-6d8b4d89878a\") " pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:11:24.909981 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.909953    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pdtks"]
Jul 28 17:11:24.913119 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:24.913087    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51709ba2_21ed_49d5_8a4f_f4c2b1f92745.slice/crio-32b19fe03ae95c22f71c324c46e27c45a9b3006b351be8b98f6052fcf9577514 WatchSource:0}: Error finding container 32b19fe03ae95c22f71c324c46e27c45a9b3006b351be8b98f6052fcf9577514: Status 404 returned error can't find the container with id 32b19fe03ae95c22f71c324c46e27c45a9b3006b351be8b98f6052fcf9577514
Jul 28 17:11:24.932048 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.932024    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dww89"]
Jul 28 17:11:24.934551 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:24.934530    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ed84b2_9963_498c_8594_54462ee49011.slice/crio-de732acab47602c329477e982e373c876144710f208027f014cc197803927cef WatchSource:0}: Error finding container de732acab47602c329477e982e373c876144710f208027f014cc197803927cef: Status 404 returned error can't find the container with id de732acab47602c329477e982e373c876144710f208027f014cc197803927cef
Jul 28 17:11:24.939699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.939678    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-22wdg"]
Jul 28 17:11:24.944021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.944003    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:24.946421 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.946405    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:24.946556 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.946540    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:24.946729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.946713    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:24.947096 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.947081    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:24.947289 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.947276    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtzhc\""
Jul 28 17:11:24.953066 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.953047    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22wdg"]
Jul 28 17:11:24.977092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:24.977060    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.008163 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.008134    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f763302e-899d-450b-a3fb-4e29fc13f2ff-data-volume\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.008283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.008172    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.008283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.008205    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhng\" (UniqueName: \"kubernetes.io/projected/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-api-access-wfhng\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.008283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.008267    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f763302e-899d-450b-a3fb-4e29fc13f2ff-crio-socket\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.008393 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.008290    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.035314 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.035253    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:25.080121 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.080100    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wz9mb\""
Jul 28 17:11:25.087898 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.087879    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:11:25.108819 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.108795    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.108922 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.108835    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhng\" (UniqueName: \"kubernetes.io/projected/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-api-access-wfhng\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.108922 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.108887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f763302e-899d-450b-a3fb-4e29fc13f2ff-crio-socket\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.108922 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.108916    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:25.109075 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.108985    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:25.608963971 +0000 UTC m=+66.161273856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:25.109075 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.109015    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:25.109075 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.108917    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.109075 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.109063    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:25.609050895 +0000 UTC m=+66.161360765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:25.109257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.109093    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f763302e-899d-450b-a3fb-4e29fc13f2ff-data-volume\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.109257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.109143    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f763302e-899d-450b-a3fb-4e29fc13f2ff-crio-socket\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.109383 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.109367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f763302e-899d-450b-a3fb-4e29fc13f2ff-data-volume\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.117859 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.117831    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhng\" (UniqueName: \"kubernetes.io/projected/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-api-access-wfhng\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.213296 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.213266    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g4wzj"]
Jul 28 17:11:25.216207 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:25.216172    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1de778_9e38_4e7d_927e_6d8b4d89878a.slice/crio-a441ab991e739cf8c1b1b082a38773bb2b75f5b8f47164d531464e9192540a3b WatchSource:0}: Error finding container a441ab991e739cf8c1b1b082a38773bb2b75f5b8f47164d531464e9192540a3b: Status 404 returned error can't find the container with id a441ab991e739cf8c1b1b082a38773bb2b75f5b8f47164d531464e9192540a3b
Jul 28 17:11:25.240319 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.240280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dww89" event={"ID":"52ed84b2-9963-498c-8594-54462ee49011","Type":"ContainerStarted","Data":"de732acab47602c329477e982e373c876144710f208027f014cc197803927cef"}
Jul 28 17:11:25.241343 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.241321    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pdtks" event={"ID":"51709ba2-21ed-49d5-8a4f-f4c2b1f92745","Type":"ContainerStarted","Data":"32b19fe03ae95c22f71c324c46e27c45a9b3006b351be8b98f6052fcf9577514"}
Jul 28 17:11:25.242405 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.242377    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4wzj" event={"ID":"3d1de778-9e38-4e7d-927e-6d8b4d89878a","Type":"ContainerStarted","Data":"a441ab991e739cf8c1b1b082a38773bb2b75f5b8f47164d531464e9192540a3b"}
Jul 28 17:11:25.531150 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.531054    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nspz_2019a327-9b47-402f-9579-4015f8476743/dns/0.log"
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.613540    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.613597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.613687    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.613748    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:26.613728412 +0000 UTC m=+67.166038285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.614178    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:25.614283 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:25.614231    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:26.614214841 +0000 UTC m=+67.166524716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:25.731858 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:25.731682    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nspz_2019a327-9b47-402f-9579-4015f8476743/kube-rbac-proxy/0.log"
Jul 28 17:11:26.113549 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.113482    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.210747 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.210658    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.308903 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.308739    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.408427 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.408304    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.623538 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.623504    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:26.623720 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.623600    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:26.623793 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:26.623757    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:26.623842 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:26.623825    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:28.623805014 +0000 UTC m=+69.176114902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:26.624002 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:26.623965    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:26.624141 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:26.624039    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:28.624024272 +0000 UTC m=+69.176334146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:26.721932 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.721781    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.732694 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.732495    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dx6mc_09870d81-7076-49f2-a276-6530e16cf472/dns-node-resolver/0.log"
Jul 28 17:11:26.821749 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.821701    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:26.914598 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:26.914541    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.003861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.003784    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.140400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.140364    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.253073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.253019    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dww89" event={"ID":"52ed84b2-9963-498c-8594-54462ee49011","Type":"ContainerStarted","Data":"7d90027c1050a82d70e26816a7715b2d0412fa7055d9f72704dadcfd243ca4b6"}
Jul 28 17:11:27.268293 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.268213    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.355519 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.355481    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.453709 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.453674    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.511673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.511642    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.531114 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.530958    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d5779b67c-555dc_390ef382-926e-4dde-a1c5-874ccf6cdf93/registry/0.log"
Jul 28 17:11:27.779699 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.779646    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.846127 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.846018    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:27.930289 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.930130    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l8bj2_b44d39e4-6f0e-43ba-962f-14f740a7bbfe/node-ca/0.log"
Jul 28 17:11:27.978589 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:27.978558    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.043737 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.043698    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.326019 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.325968    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.388110 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.388018    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.444208 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.444170    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:28.530673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.530627    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cm88p_ed188f2a-8cb8-47dd-9389-e46104e0de0b/serve-healthcheck-canary/0.log"
Jul 28 17:11:28.640169 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.640085    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:28.640399 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:28.640191    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:28.640399 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:28.640282    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:28.640399 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:28.640305    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:28.640399 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:28.640357    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:32.640338582 +0000 UTC m=+73.192648457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:28.640399 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:28.640378    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:32.640368054 +0000 UTC m=+73.192677926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:29.137072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:29.137035    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:29.193191 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:29.193159    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.445941 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:30.445907    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:30.546032 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:30.545985    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.386196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:31.386148    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:31.484625 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:31.484591    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.273673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.273633    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dww89" event={"ID":"52ed84b2-9963-498c-8594-54462ee49011","Type":"ContainerStarted","Data":"84ee11c4efed0ca724a60e0be1509f042c7b1c3641212192396c234cf88fa5bc"}
Jul 28 17:11:32.277373 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.277340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pdtks" event={"ID":"51709ba2-21ed-49d5-8a4f-f4c2b1f92745","Type":"ContainerStarted","Data":"ec39467cce39569fc78b13e144446dcccc1559a5164bf122e1831ad47ff481f7"}
Jul 28 17:11:32.281423 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.281396    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g4wzj" event={"ID":"3d1de778-9e38-4e7d-927e-6d8b4d89878a","Type":"ContainerStarted","Data":"7788c91c8f6631d518cbb45566db9690fb8909dd8830b017a0deaeca04ed0e60"}
Jul 28 17:11:32.281543 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.281449    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:11:32.283225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.283150    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2" event={"ID":"882bf7f0-d231-42de-b824-9a4c756658f6","Type":"ContainerStarted","Data":"db76e609422aac0b0f8ed817fae7b2f0a7385b51d45e4f13672545fbe49108ab"}
Jul 28 17:11:32.283865 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.283839    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:32.285662 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.285638    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2"
Jul 28 17:11:32.285768 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.285745    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc" event={"ID":"bcb2c83f-6bab-486f-a5d5-e4cba341d168","Type":"ContainerStarted","Data":"3f86d52ae29508d46f7ee290a0c7264a9021bed96231b0e48d6eb3bd3b0253b4"}
Jul 28 17:11:32.292757 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.292708    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dww89" podStartSLOduration=70.998538892 podStartE2EDuration="1m12.292693083s" totalImagesPullingTime="1.294154191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:32.291923694 +0000 UTC m=+72.844233584" watchObservedRunningTime="2026-07-28 17:11:32.292693083 +0000 UTC m=+72.845002974"
Jul 28 17:11:32.310444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.310400    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-847c7cd69d-gjmb2" podStartSLOduration=1.178989152 podStartE2EDuration="11.310385344s" totalImagesPullingTime="10.131396192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:32.309391831 +0000 UTC m=+72.861701723" watchObservedRunningTime="2026-07-28 17:11:32.310385344 +0000 UTC m=+72.862695235"
Jul 28 17:11:32.327154 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.327093    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g4wzj" podStartSLOduration=65.446993555 podStartE2EDuration="1m12.327072885s" totalImagesPullingTime="6.88007933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:32.326298789 +0000 UTC m=+72.878608682" watchObservedRunningTime="2026-07-28 17:11:32.327072885 +0000 UTC m=+72.879382777"
Jul 28 17:11:32.359388 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.359283    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c5d44656b-fc7sc" podStartSLOduration=1.203975568 podStartE2EDuration="11.359268919s" totalImagesPullingTime="10.155293351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:32.35923525 +0000 UTC m=+72.911545161" watchObservedRunningTime="2026-07-28 17:11:32.359268919 +0000 UTC m=+72.911578810"
Jul 28 17:11:32.359540 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.359454    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pdtks" podStartSLOduration=65.163761449 podStartE2EDuration="1m12.359442743s" totalImagesPullingTime="7.195681294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:32.343110392 +0000 UTC m=+72.895420284" watchObservedRunningTime="2026-07-28 17:11:32.359442743 +0000 UTC m=+72.911752636"
Jul 28 17:11:32.394505 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.394476    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.497924 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.497893    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:32.670470 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.670395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:32.670470 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:32.670433    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:32.670675 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:32.670513    2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:32.670675 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:32.670557    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:32.670675 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:32.670568    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:40.670554432 +0000 UTC m=+81.222864300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:32.670675 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:32.670630    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls podName:f763302e-899d-450b-a3fb-4e29fc13f2ff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:40.670611068 +0000 UTC m=+81.222920951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-22wdg" (UID: "f763302e-899d-450b-a3fb-4e29fc13f2ff") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:33.937426 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:33.937390    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:34.002974 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:34.002925    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:34.706411 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:34.706375    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.100107 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.100073    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.196703 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.196675    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.263437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.263405    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.326544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.326509    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.425757 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.425688    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.526328 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.526298    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:36.628231 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:36.628191    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:37.485710 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:37.485668    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:37.714596 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:37.714567    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.096518 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.096482    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.304413 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.304383    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.366965 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.366864    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zmzxb"]
Jul 28 17:11:39.370734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.370719    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.373604 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.373582    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:11:39.373736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.373674    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:11:39.374398 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.374383    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:11:39.374626 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.374613    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:11:39.375218 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.375200    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:11:39.375510 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.375497    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:11:39.375705 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.375690    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c44q6\""
Jul 28 17:11:39.410384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.410360    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.421282 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421261    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-wtmp\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421391 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421308    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421391 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421364    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-metrics-client-ca\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421477 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421404    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22d7d\" (UniqueName: \"kubernetes.io/projected/383f0e38-a7c2-4981-803c-e89f3258efcd-kube-api-access-22d7d\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421477 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421450    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421474    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-textfile\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421515    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-root\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421551    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-sys\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.421694 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.421584    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.512330 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.512299    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.521919 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.521895    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-root\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522065 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.521937    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-sys\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522065 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.521956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522065 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522027    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-wtmp\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522067    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522027    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-sys\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522028    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-root\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522129    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-wtmp\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-metrics-client-ca\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:39.522159    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522170    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22d7d\" (UniqueName: \"kubernetes.io/projected/383f0e38-a7c2-4981-803c-e89f3258efcd-kube-api-access-22d7d\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522230 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:39.522217    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls podName:383f0e38-a7c2-4981-803c-e89f3258efcd nodeName:}" failed. No retries permitted until 2026-07-28 17:11:40.022197698 +0000 UTC m=+80.574507583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls") pod "node-exporter-zmzxb" (UID: "383f0e38-a7c2-4981-803c-e89f3258efcd") : secret "node-exporter-tls" not found
Jul 28 17:11:39.522618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522237    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522273    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-textfile\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522543    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-textfile\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522758 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522710    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-metrics-client-ca\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.522758 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.522723    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.524212 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.524190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.531634 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.531611    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22d7d\" (UniqueName: \"kubernetes.io/projected/383f0e38-a7c2-4981-803c-e89f3258efcd-kube-api-access-22d7d\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:39.612296 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.612266    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.714615 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.714588    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:39.817257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:39.817228    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.026207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.026126    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:40.028336 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.028313    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/383f0e38-a7c2-4981-803c-e89f3258efcd-node-exporter-tls\") pod \"node-exporter-zmzxb\" (UID: \"383f0e38-a7c2-4981-803c-e89f3258efcd\") " pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:40.280120 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.280035    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zmzxb"
Jul 28 17:11:40.292002 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:40.291960    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383f0e38_a7c2_4981_803c_e89f3258efcd.slice/crio-461649c65960943a0c6c7195546fcc46c67d87e0d226b38ba78ad079b6f368ba WatchSource:0}: Error finding container 461649c65960943a0c6c7195546fcc46c67d87e0d226b38ba78ad079b6f368ba: Status 404 returned error can't find the container with id 461649c65960943a0c6c7195546fcc46c67d87e0d226b38ba78ad079b6f368ba
Jul 28 17:11:40.311919 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.311883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmzxb" event={"ID":"383f0e38-a7c2-4981-803c-e89f3258efcd","Type":"ContainerStarted","Data":"461649c65960943a0c6c7195546fcc46c67d87e0d226b38ba78ad079b6f368ba"}
Jul 28 17:11:40.348324 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.348294    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.448215 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.448190    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.454536 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.454512    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:40.457825 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.457806    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.461752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461730    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:11:40.461875 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461856    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:11:40.461971 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461864    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:11:40.461971 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461919    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lbjj6\""
Jul 28 17:11:40.461971 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461953    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:11:40.461971 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.461984    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:11:40.462269 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.462181    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:11:40.462269 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.462211    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:11:40.462562 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.462547    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:11:40.463561 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.463544    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:11:40.479467 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.479442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:40.529687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529660    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529689    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529867 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529707    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529867 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s86z\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529867 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529825    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529885    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.529976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529939    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.530146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.529978    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.530146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.530015    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.530146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.530085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.530146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.530103    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.530146 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.530123    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.550269 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.550242    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.630807 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630778    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.630807 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630809    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.630957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630827    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.630957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630848    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631059 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630953    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631059 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.630978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631059 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631040    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s86z\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631202 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631077    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631202 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:40.631098    2552 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 28 17:11:40.631202 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631125    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631202 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631164    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631202 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631461 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631461 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631249    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631461 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.631255    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.631673 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:40.631651    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls podName:be9835d0-f466-4a52-8d17-c1bad4c94aff nodeName:}" failed. No retries permitted until 2026-07-28 17:11:41.131627147 +0000 UTC m=+81.683937029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff") : secret "alertmanager-main-tls" not found
Jul 28 17:11:40.632240 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.632209    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.632599 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.632531    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634110 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634087    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634213 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634112    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634213 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634087    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634567 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634526    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634660 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634587    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.634724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.634710    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.635089 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.635059    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.635489 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.635417    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.640552 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.640536    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s86z\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:40.731970 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.731939    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:40.732121 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.731979    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:40.732460 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.732435    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f763302e-899d-450b-a3fb-4e29fc13f2ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:40.734103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.734081    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f763302e-899d-450b-a3fb-4e29fc13f2ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-22wdg\" (UID: \"f763302e-899d-450b-a3fb-4e29fc13f2ff\") " pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:40.758881 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.758851    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:40.853387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:40.853318    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-22wdg"
Jul 28 17:11:41.007600 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.007493    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-22wdg"]
Jul 28 17:11:41.138661 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.138575    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:41.141524 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.141496    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:41.201555 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:41.201524    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf763302e_899d_450b_a3fb_4e29fc13f2ff.slice/crio-d8e274fa0200ec0733d3287603e453b991b1bc36f488b2b4c697c735105461d2 WatchSource:0}: Error finding container d8e274fa0200ec0733d3287603e453b991b1bc36f488b2b4c697c735105461d2: Status 404 returned error can't find the container with id d8e274fa0200ec0733d3287603e453b991b1bc36f488b2b4c697c735105461d2
Jul 28 17:11:41.317057 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.317020    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmzxb" event={"ID":"383f0e38-a7c2-4981-803c-e89f3258efcd","Type":"ContainerStarted","Data":"c052f43fcb4324e6a0f9e332393aa8578c7c1cda236fe5dbab867ca69540de50"}
Jul 28 17:11:41.318443 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.318418    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22wdg" event={"ID":"f763302e-899d-450b-a3fb-4e29fc13f2ff","Type":"ContainerStarted","Data":"d8e274fa0200ec0733d3287603e453b991b1bc36f488b2b4c697c735105461d2"}
Jul 28 17:11:41.366688 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.366650    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:41.501029 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.500884    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:41.502896 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:41.502871    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9835d0_f466_4a52_8d17_c1bad4c94aff.slice/crio-971ece80e431f95d1de4510caf249cb84b30ba96237d75355ddf914969fe6b56 WatchSource:0}: Error finding container 971ece80e431f95d1de4510caf249cb84b30ba96237d75355ddf914969fe6b56: Status 404 returned error can't find the container with id 971ece80e431f95d1de4510caf249cb84b30ba96237d75355ddf914969fe6b56
Jul 28 17:11:41.606349 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:41.606299    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.286778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.286657    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.324935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.324901    2552 generic.go:356] "Generic (PLEG): container finished" podID="383f0e38-a7c2-4981-803c-e89f3258efcd" containerID="c052f43fcb4324e6a0f9e332393aa8578c7c1cda236fe5dbab867ca69540de50" exitCode=0
Jul 28 17:11:42.325401 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.325015    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmzxb" event={"ID":"383f0e38-a7c2-4981-803c-e89f3258efcd","Type":"ContainerDied","Data":"c052f43fcb4324e6a0f9e332393aa8578c7c1cda236fe5dbab867ca69540de50"}
Jul 28 17:11:42.327311 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.326783    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"971ece80e431f95d1de4510caf249cb84b30ba96237d75355ddf914969fe6b56"}
Jul 28 17:11:42.376895 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.376807    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.441799 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.441740    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.598203 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:42.598125    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.333044 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.332642    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmzxb" event={"ID":"383f0e38-a7c2-4981-803c-e89f3258efcd","Type":"ContainerStarted","Data":"7d6a19f93c230cad9f161eb23ca2d9f3aba40b16a5163e0b3be009dbd85f309a"}
Jul 28 17:11:43.333044 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.332706    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmzxb" event={"ID":"383f0e38-a7c2-4981-803c-e89f3258efcd","Type":"ContainerStarted","Data":"d09a90850496eb63411ed0b5c4d8bb307dead47177f8dd3bc9806d4951ab81f5"}
Jul 28 17:11:43.334541 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.334510    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35" exitCode=0
Jul 28 17:11:43.334690 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.334590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"}
Jul 28 17:11:43.355599 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.355553    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zmzxb" podStartSLOduration=3.341094175 podStartE2EDuration="4.3555384s" totalImagesPullingTime="951.001999ms" totalInitContainerRuntime="63.442226ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:43.354688763 +0000 UTC m=+83.906998659" watchObservedRunningTime="2026-07-28 17:11:43.3555384 +0000 UTC m=+83.907848312"
Jul 28 17:11:43.805855 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.805795    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:43.871060 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:43.871018    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.121314 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.121230    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.121617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.121590    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"]
Jul 28 17:11:44.138053 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.138028    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"]
Jul 28 17:11:44.138220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.138105    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.141123 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.141103    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:11:44.141251 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.141153    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4pgnh\""
Jul 28 17:11:44.181794 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.181757    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.239953 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.239926    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d5779b67c-555dc"
Jul 28 17:11:44.267068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.267037    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.267219 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.267119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqf2\" (UniqueName: \"kubernetes.io/projected/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-kube-api-access-mdqf2\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.368143 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.368100    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqf2\" (UniqueName: \"kubernetes.io/projected/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-kube-api-access-mdqf2\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.368600 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.368240    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.368600 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:44.368520    2552 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 17:11:44.368600 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:11:44.368586    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert podName:0837ee53-b9ed-4ff2-93e7-f272e30fcb82 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:44.868565651 +0000 UTC m=+85.420875534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-6tzs4" (UID: "0837ee53-b9ed-4ff2-93e7-f272e30fcb82") : secret "monitoring-plugin-cert" not found
Jul 28 17:11:44.380610 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.380543    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqf2\" (UniqueName: \"kubernetes.io/projected/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-kube-api-access-mdqf2\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.400494 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.400454    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.501471 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.501436    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.559624 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.559584    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.658508 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.658433    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:44.871450 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.871418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:44.874068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:44.874044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0837ee53-b9ed-4ff2-93e7-f272e30fcb82-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-6tzs4\" (UID: \"0837ee53-b9ed-4ff2-93e7-f272e30fcb82\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:45.048133 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:45.048105    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:45.655874 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:45.655822    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:46.255241 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:46.255209    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"]
Jul 28 17:11:46.491358 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:11:46.491327    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0837ee53_b9ed_4ff2_93e7_f272e30fcb82.slice/crio-fb961bebb27c5a08121a9dad95c94ec65a27369ef71b1fca256b6b2d8632d469 WatchSource:0}: Error finding container fb961bebb27c5a08121a9dad95c94ec65a27369ef71b1fca256b6b2d8632d469: Status 404 returned error can't find the container with id fb961bebb27c5a08121a9dad95c94ec65a27369ef71b1fca256b6b2d8632d469
Jul 28 17:11:47.360906 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.360869    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22wdg" event={"ID":"f763302e-899d-450b-a3fb-4e29fc13f2ff","Type":"ContainerStarted","Data":"2fb2ddd9841cacf942065862d46a053990f848d9fc452efebe74049b1fadca99"}
Jul 28 17:11:47.362454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.362414    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4" event={"ID":"0837ee53-b9ed-4ff2-93e7-f272e30fcb82","Type":"ContainerStarted","Data":"fb961bebb27c5a08121a9dad95c94ec65a27369ef71b1fca256b6b2d8632d469"}
Jul 28 17:11:47.366694 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.366670    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"}
Jul 28 17:11:47.366814 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.366727    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"}
Jul 28 17:11:47.366814 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.366753    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"}
Jul 28 17:11:47.366814 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.366778    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"}
Jul 28 17:11:47.366814 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:47.366800    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"}
Jul 28 17:11:48.373081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.372968    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4" event={"ID":"0837ee53-b9ed-4ff2-93e7-f272e30fcb82","Type":"ContainerStarted","Data":"38914f4dc87e8c3403e0055fee50c4d9df1576dc18c5d4c300a0743c2b2b2183"}
Jul 28 17:11:48.373081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.373053    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:48.377212 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.377180    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerStarted","Data":"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"}
Jul 28 17:11:48.379169 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.379143    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4"
Jul 28 17:11:48.395240 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.395191    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-6tzs4" podStartSLOduration=2.780333138 podStartE2EDuration="4.395177598s" totalImagesPullingTime="1.61484446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:48.394701241 +0000 UTC m=+88.947011138" watchObservedRunningTime="2026-07-28 17:11:48.395177598 +0000 UTC m=+88.947487490"
Jul 28 17:11:48.431544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:48.431494    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.704765916 podStartE2EDuration="8.43147825s" totalImagesPullingTime="5.705100607s" totalInitContainerRuntime="21.611727ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:48.428781008 +0000 UTC m=+88.981090902" watchObservedRunningTime="2026-07-28 17:11:48.43147825 +0000 UTC m=+88.983788140"
Jul 28 17:11:49.387486 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:49.387447    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.467857 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:49.467826    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.392483 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:51.392443    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22wdg" event={"ID":"f763302e-899d-450b-a3fb-4e29fc13f2ff","Type":"ContainerStarted","Data":"dfab386fef6ce146ac1d4cb22e952583fa4c32c782a5536a7f03e5a2acb62cd3"}
Jul 28 17:11:54.405122 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:54.405090    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-22wdg" event={"ID":"f763302e-899d-450b-a3fb-4e29fc13f2ff","Type":"ContainerStarted","Data":"0666f055003630210b321011165340bce7e548b51d084564e133a908b2c19ea9"}
Jul 28 17:11:54.425282 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:54.425239    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-22wdg" podStartSLOduration=17.969398646 podStartE2EDuration="30.425224209s" totalImagesPullingTime="12.455825563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:54.423806436 +0000 UTC m=+94.976116347" watchObservedRunningTime="2026-07-28 17:11:54.425224209 +0000 UTC m=+94.977534099"
Jul 28 17:11:55.419491 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:55.419444    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:55.496716 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:55.496687    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.180937 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:56.180905    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.248257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:56.248222    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.852860 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:56.852818    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:56.939078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:56.939049    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.140845 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:59.140803    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:59.250959 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:11:59.250924    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:03.291806 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:03.291772    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g4wzj"
Jul 28 17:12:14.149248 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:14.149196    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:14.213503 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:14.213476    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:23.235072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:23.235038    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:29.879580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:29.879547    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:29.932198 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:29.932171    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:55.178347 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:55.178311    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:55.281025 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:55.280981    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:58.572462 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:58.572424    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:58.803843 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:58.803814    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:59.619593 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.619561    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:12:59.619980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.619844    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="alertmanager" containerID="cri-o://a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa" gracePeriod=120
Jul 28 17:12:59.619980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.619897    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-metric" containerID="cri-o://3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d" gracePeriod=120
Jul 28 17:12:59.619980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.619932    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-web" containerID="cri-o://f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028" gracePeriod=120
Jul 28 17:12:59.620174 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.619982    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy" containerID="cri-o://5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693" gracePeriod=120
Jul 28 17:12:59.620174 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.620047    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="prom-label-proxy" containerID="cri-o://755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03" gracePeriod=120
Jul 28 17:12:59.620174 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:12:59.620008    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="config-reloader" containerID="cri-o://429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229" gracePeriod=120
Jul 28 17:13:00.640637 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640604    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03" exitCode=0
Jul 28 17:13:00.640637 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640632    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693" exitCode=0
Jul 28 17:13:00.640637 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640639    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229" exitCode=0
Jul 28 17:13:00.640637 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640645    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa" exitCode=0
Jul 28 17:13:00.641118 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640672    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"}
Jul 28 17:13:00.641118 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640738    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"}
Jul 28 17:13:00.641118 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640764    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"}
Jul 28 17:13:00.641118 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.640790    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"}
Jul 28 17:13:00.864860 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.864833    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:00.917966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.917896    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.917966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.917929    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.917966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.917951    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s86z\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.917973    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918018    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918056    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918089    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918117    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918163    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918194    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918221    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918258    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918732 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918291    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web\") pod \"be9835d0-f466-4a52-8d17-c1bad4c94aff\" (UID: \"be9835d0-f466-4a52-8d17-c1bad4c94aff\") "
Jul 28 17:13:00.918732 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.918613    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:00.920151 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.920115    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:00.920349 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.920326    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:00.922244 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.922166    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume" (OuterVolumeSpecName: "config-volume") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.922244 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.922173    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z" (OuterVolumeSpecName: "kube-api-access-5s86z") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "kube-api-access-5s86z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:00.922642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.922599    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.922900 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.922840    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out" (OuterVolumeSpecName: "config-out") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:13:00.922900 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.922858    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.923078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.923058    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:00.923297 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.923263    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.923667 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.923639    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.926168 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.926139    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:00.932454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:00.932431    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config" (OuterVolumeSpecName: "web-config") pod "be9835d0-f466-4a52-8d17-c1bad4c94aff" (UID: "be9835d0-f466-4a52-8d17-c1bad4c94aff"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:01.018746 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018716    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-cluster-tls-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018746 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018743    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-tls-assets\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018756    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s86z\" (UniqueName: \"kubernetes.io/projected/be9835d0-f466-4a52-8d17-c1bad4c94aff-kube-api-access-5s86z\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018792    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-web-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018806    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-volume\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018817    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-main-db\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018831    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018845    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-main-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018858    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-metrics-client-ca\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018870    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018883    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9835d0-f466-4a52-8d17-c1bad4c94aff-config-out\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018894    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9835d0-f466-4a52-8d17-c1bad4c94aff-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.018944 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.018907    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9835d0-f466-4a52-8d17-c1bad4c94aff-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:13:01.647053 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647018    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d" exitCode=0
Jul 28 17:13:01.647053 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647042    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerID="f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028" exitCode=0
Jul 28 17:13:01.647490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647084    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"}
Jul 28 17:13:01.647490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647092    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.647490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647142    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"}
Jul 28 17:13:01.647490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647172    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9835d0-f466-4a52-8d17-c1bad4c94aff","Type":"ContainerDied","Data":"971ece80e431f95d1de4510caf249cb84b30ba96237d75355ddf914969fe6b56"}
Jul 28 17:13:01.647490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.647202    2552 scope.go:117] "RemoveContainer" containerID="755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"
Jul 28 17:13:01.654833 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.654814    2552 scope.go:117] "RemoveContainer" containerID="3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"
Jul 28 17:13:01.661427 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.661408    2552 scope.go:117] "RemoveContainer" containerID="5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"
Jul 28 17:13:01.667687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.667670    2552 scope.go:117] "RemoveContainer" containerID="f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"
Jul 28 17:13:01.675675 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.675643    2552 scope.go:117] "RemoveContainer" containerID="429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"
Jul 28 17:13:01.677673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.677653    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:01.682668 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.682646    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:01.683580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.683562    2552 scope.go:117] "RemoveContainer" containerID="a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"
Jul 28 17:13:01.690570 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.690550    2552 scope.go:117] "RemoveContainer" containerID="cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"
Jul 28 17:13:01.696897 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.696882    2552 scope.go:117] "RemoveContainer" containerID="755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"
Jul 28 17:13:01.697166 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.697135    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03\": container with ID starting with 755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03 not found: ID does not exist" containerID="755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"
Jul 28 17:13:01.697246 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697174    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"} err="failed to get container status \"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03\": rpc error: code = NotFound desc = could not find container \"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03\": container with ID starting with 755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03 not found: ID does not exist"
Jul 28 17:13:01.697246 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697207    2552 scope.go:117] "RemoveContainer" containerID="3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"
Jul 28 17:13:01.697420 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.697402    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d\": container with ID starting with 3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d not found: ID does not exist" containerID="3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"
Jul 28 17:13:01.697476 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697426    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"} err="failed to get container status \"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d\": rpc error: code = NotFound desc = could not find container \"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d\": container with ID starting with 3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d not found: ID does not exist"
Jul 28 17:13:01.697476 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697444    2552 scope.go:117] "RemoveContainer" containerID="5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"
Jul 28 17:13:01.697676 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.697659    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693\": container with ID starting with 5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693 not found: ID does not exist" containerID="5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"
Jul 28 17:13:01.697735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697681    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"} err="failed to get container status \"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693\": rpc error: code = NotFound desc = could not find container \"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693\": container with ID starting with 5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693 not found: ID does not exist"
Jul 28 17:13:01.697735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697695    2552 scope.go:117] "RemoveContainer" containerID="f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"
Jul 28 17:13:01.697916 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.697901    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028\": container with ID starting with f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028 not found: ID does not exist" containerID="f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"
Jul 28 17:13:01.697969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697919    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"} err="failed to get container status \"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028\": rpc error: code = NotFound desc = could not find container \"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028\": container with ID starting with f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028 not found: ID does not exist"
Jul 28 17:13:01.697969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.697931    2552 scope.go:117] "RemoveContainer" containerID="429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"
Jul 28 17:13:01.698137 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.698119    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229\": container with ID starting with 429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229 not found: ID does not exist" containerID="429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"
Jul 28 17:13:01.698188 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698141    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"} err="failed to get container status \"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229\": rpc error: code = NotFound desc = could not find container \"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229\": container with ID starting with 429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229 not found: ID does not exist"
Jul 28 17:13:01.698188 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698155    2552 scope.go:117] "RemoveContainer" containerID="a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"
Jul 28 17:13:01.698602 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.698361    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa\": container with ID starting with a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa not found: ID does not exist" containerID="a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"
Jul 28 17:13:01.698602 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698377    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"} err="failed to get container status \"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa\": rpc error: code = NotFound desc = could not find container \"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa\": container with ID starting with a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa not found: ID does not exist"
Jul 28 17:13:01.698602 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698390    2552 scope.go:117] "RemoveContainer" containerID="cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"
Jul 28 17:13:01.698602 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:13:01.698591    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35\": container with ID starting with cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35 not found: ID does not exist" containerID="cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"
Jul 28 17:13:01.698894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698611    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"} err="failed to get container status \"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35\": rpc error: code = NotFound desc = could not find container \"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35\": container with ID starting with cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35 not found: ID does not exist"
Jul 28 17:13:01.698894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698627    2552 scope.go:117] "RemoveContainer" containerID="755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"
Jul 28 17:13:01.698894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698825    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03"} err="failed to get container status \"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03\": rpc error: code = NotFound desc = could not find container \"755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03\": container with ID starting with 755ebd827deaa43ed5a1c9998cd61929ed724c411be35099775d6ae9ab047d03 not found: ID does not exist"
Jul 28 17:13:01.698894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.698840    2552 scope.go:117] "RemoveContainer" containerID="3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"
Jul 28 17:13:01.699196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699054    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d"} err="failed to get container status \"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d\": rpc error: code = NotFound desc = could not find container \"3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d\": container with ID starting with 3faaf7628fb4ea2133d9e7c80a6ddd48ce3bdf14f9e7eb41e8246c849b24c36d not found: ID does not exist"
Jul 28 17:13:01.699196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699074    2552 scope.go:117] "RemoveContainer" containerID="5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"
Jul 28 17:13:01.699444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699309    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693"} err="failed to get container status \"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693\": rpc error: code = NotFound desc = could not find container \"5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693\": container with ID starting with 5e87f4c643821fbb24450f54c452dfdd634f0b4c84806cc488d8bf6f459d2693 not found: ID does not exist"
Jul 28 17:13:01.699444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699336    2552 scope.go:117] "RemoveContainer" containerID="f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"
Jul 28 17:13:01.699599 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699579    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028"} err="failed to get container status \"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028\": rpc error: code = NotFound desc = could not find container \"f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028\": container with ID starting with f20d41860be1f3678d40a7f73ef70960c954ea094b741182a8be8739f4911028 not found: ID does not exist"
Jul 28 17:13:01.699677 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699600    2552 scope.go:117] "RemoveContainer" containerID="429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"
Jul 28 17:13:01.699839 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699811    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229"} err="failed to get container status \"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229\": rpc error: code = NotFound desc = could not find container \"429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229\": container with ID starting with 429d8a3b382fdd7249e1d68da1d2816bb6486b70306618979fa45c4ea4ca7229 not found: ID does not exist"
Jul 28 17:13:01.699839 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.699836    2552 scope.go:117] "RemoveContainer" containerID="a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"
Jul 28 17:13:01.700127 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.700107    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa"} err="failed to get container status \"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa\": rpc error: code = NotFound desc = could not find container \"a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa\": container with ID starting with a53be2dc1f5e3292024d19b15aa5faff3f2e8c43e687814459de8aa4b904bafa not found: ID does not exist"
Jul 28 17:13:01.700197 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.700127    2552 scope.go:117] "RemoveContainer" containerID="cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"
Jul 28 17:13:01.700381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.700347    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35"} err="failed to get container status \"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35\": rpc error: code = NotFound desc = could not find container \"cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35\": container with ID starting with cdfdd0d44c615a65e7ac71ad1893c9a3a012826e246e2083f44bf455cc1a9c35 not found: ID does not exist"
Jul 28 17:13:01.723873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.723844    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:01.724109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724096    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="config-reloader"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724110    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="config-reloader"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724121    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724127    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724135    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="init-config-reloader"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724142    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="init-config-reloader"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724147    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-web"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724152    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-web"
Jul 28 17:13:01.724156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724158    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="prom-label-proxy"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724164    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="prom-label-proxy"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724171    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724175    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724182    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="alertmanager"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724186    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="alertmanager"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724220    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724228    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="prom-label-proxy"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724233    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-web"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724238    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="kube-rbac-proxy-metric"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724245    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="alertmanager"
Jul 28 17:13:01.724385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.724250    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" containerName="config-reloader"
Jul 28 17:13:01.728710 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.728694    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.732171 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732150    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:13:01.732273 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732185    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:13:01.732273 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732194    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:13:01.732520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732502    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:13:01.732621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732601    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:13:01.732762 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732747    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:13:01.732762 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732749    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:13:01.732868 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.732814    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:13:01.733091 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.733077    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lbjj6\""
Jul 28 17:13:01.738490 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.738473    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:13:01.741424 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.741404    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:01.824278 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824248    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh65w\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-kube-api-access-dh65w\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824278 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-config-out\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824363    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824405    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824579 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824468    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824579 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824579 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824523    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824579 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824541    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-web-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824577    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824604    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.824692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.824623    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.925803 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925716    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.925803 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925761    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-web-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.925803 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.925803 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925821    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh65w\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-kube-api-access-dh65w\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925873    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925912    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-config-out\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.925970    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.926015    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.926050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926116 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.926083    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926540 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.926297    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.926812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.926780    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.927521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.927211    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f290579-5839-4bf9-8876-1673e4f3c347-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929056 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.928860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929056 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.928916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-web-config\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929056 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.928945    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929248 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.929070    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929248 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.929169    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9f290579-5839-4bf9-8876-1673e4f3c347-config-out\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929451 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.929427    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929611 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.929593    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-config-volume\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.929806 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.929789    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f290579-5839-4bf9-8876-1673e4f3c347-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.930761 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.930736    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.936081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.936059    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh65w\" (UniqueName: \"kubernetes.io/projected/9f290579-5839-4bf9-8876-1673e4f3c347-kube-api-access-dh65w\") pod \"alertmanager-main-0\" (UID: \"9f290579-5839-4bf9-8876-1673e4f3c347\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:01.970941 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:01.970919    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9835d0-f466-4a52-8d17-c1bad4c94aff" path="/var/lib/kubelet/pods/be9835d0-f466-4a52-8d17-c1bad4c94aff/volumes"
Jul 28 17:13:02.037554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:02.037531    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:13:02.201796 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:02.201768    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:13:02.202213 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:13:02.202192    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f290579_5839_4bf9_8876_1673e4f3c347.slice/crio-734fc73f46cf65f211c18cd782192d89921d8ba2e4f48f81566ffeb681e8465b WatchSource:0}: Error finding container 734fc73f46cf65f211c18cd782192d89921d8ba2e4f48f81566ffeb681e8465b: Status 404 returned error can't find the container with id 734fc73f46cf65f211c18cd782192d89921d8ba2e4f48f81566ffeb681e8465b
Jul 28 17:13:02.652886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:02.652852    2552 generic.go:356] "Generic (PLEG): container finished" podID="9f290579-5839-4bf9-8876-1673e4f3c347" containerID="94f6cfc61ff1d00d9d15e829506caa60fc62e18ad99eb306b03516b3cbc608d9" exitCode=0
Jul 28 17:13:02.653310 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:02.652939    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerDied","Data":"94f6cfc61ff1d00d9d15e829506caa60fc62e18ad99eb306b03516b3cbc608d9"}
Jul 28 17:13:02.653310 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:02.653021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"734fc73f46cf65f211c18cd782192d89921d8ba2e4f48f81566ffeb681e8465b"}
Jul 28 17:13:03.663736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663699    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"9640e2b3c0222cc9ee1edd38745c6ba1d6e7bd003471f61d9783d763b227aa6c"}
Jul 28 17:13:03.664226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663806    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"3477c60779b0558a2bcd7ec4be7afe12cd16828f13d3c88b874fb0fcdd991c01"}
Jul 28 17:13:03.664226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663853    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"3a8cb3e163ecb13412103744e9b6c6238e4946d8a8107608b94bf03699d55132"}
Jul 28 17:13:03.664226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663896    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"b71154f9e2356001ac951b41325cb50dd201c96368aefaeb34d0b365ece7b15c"}
Jul 28 17:13:03.664226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663931    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"d6d298f79b702113a04f670b39a7776a59e1b99a275fbb0d7e062460184b800c"}
Jul 28 17:13:03.664226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.663976    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9f290579-5839-4bf9-8876-1673e4f3c347","Type":"ContainerStarted","Data":"bc9d76ab26f11e19c6a3cf0a0ededb167e3a1a393142fbf227d28f9eeb6aff71"}
Jul 28 17:13:03.666282 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.666256    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c5cd568-hlwqg"]
Jul 28 17:13:03.669407 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.669391    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.671949 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.671925    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:13:03.672043 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.671978    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:13:03.672162 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.672149    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:13:03.672302 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.672287    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:13:03.672433 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.672417    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:13:03.672485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.672418    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn4kj\""
Jul 28 17:13:03.677080 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.677064    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:13:03.684088 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.684069    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c5cd568-hlwqg"]
Jul 28 17:13:03.694982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.694938    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.682947809 podStartE2EDuration="2.694924847s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.977038ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:03.692776196 +0000 UTC m=+164.245086078" watchObservedRunningTime="2026-07-28 17:13:03.694924847 +0000 UTC m=+164.247234766"
Jul 28 17:13:03.740097 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740071    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740169    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-metrics-client-ca\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740192    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740413 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740237    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-serving-certs-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740413 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740314    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740413 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740372    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-federate-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.740555 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.740433    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9zt\" (UniqueName: \"kubernetes.io/projected/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-kube-api-access-zn9zt\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841435 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841401    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9zt\" (UniqueName: \"kubernetes.io/projected/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-kube-api-access-zn9zt\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841467    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841486    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841505    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-metrics-client-ca\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841520    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841540    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-serving-certs-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841567    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.841850 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.841619    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-federate-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.842536 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.842506    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.842657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.842584    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-serving-certs-ca-bundle\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.842657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.842636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-metrics-client-ca\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.844134 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.844110    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-federate-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.844282 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.844258    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.844475 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.844459    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-secret-telemeter-client\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.844576 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.844556    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-telemeter-client-tls\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.853173 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.853147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9zt\" (UniqueName: \"kubernetes.io/projected/9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd-kube-api-access-zn9zt\") pod \"telemeter-client-6c5cd568-hlwqg\" (UID: \"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd\") " pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:03.978728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:03.978692    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg"
Jul 28 17:13:04.129972 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:04.129928    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c5cd568-hlwqg"]
Jul 28 17:13:04.131875 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:13:04.131849    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa41006_46c0_4d9c_bb9f_e3dd8362b9dd.slice/crio-1b68f9c004c0567f3c01a68fc795ae6377664299fe8cd6e6b3b246a18da9a164 WatchSource:0}: Error finding container 1b68f9c004c0567f3c01a68fc795ae6377664299fe8cd6e6b3b246a18da9a164: Status 404 returned error can't find the container with id 1b68f9c004c0567f3c01a68fc795ae6377664299fe8cd6e6b3b246a18da9a164
Jul 28 17:13:04.668749 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:04.668714    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg" event={"ID":"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd","Type":"ContainerStarted","Data":"1b68f9c004c0567f3c01a68fc795ae6377664299fe8cd6e6b3b246a18da9a164"}
Jul 28 17:13:06.678969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:06.678929    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg" event={"ID":"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd","Type":"ContainerStarted","Data":"9ff7a7089dc705e286531f50bb9bf303204df36033e0926d969f9669a0932381"}
Jul 28 17:13:06.679355 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:06.679048    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg" event={"ID":"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd","Type":"ContainerStarted","Data":"2fe0ef0a6557ff73fbe7e96da571d6e062f233d82738f5007208b7e2b8722d18"}
Jul 28 17:13:06.679355 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:06.679082    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg" event={"ID":"9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd","Type":"ContainerStarted","Data":"01ca87b8c0c94d34bece961c46d05710df2ab63e0cf18eb47db5d9074999b927"}
Jul 28 17:13:06.711850 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:06.711801    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c5cd568-hlwqg" podStartSLOduration=1.661202193 podStartE2EDuration="3.711786981s" totalImagesPullingTime="2.050584788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:06.710309449 +0000 UTC m=+167.262619350" watchObservedRunningTime="2026-07-28 17:13:06.711786981 +0000 UTC m=+167.264096872"
Jul 28 17:13:07.524566 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:07.524536    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:07.583683 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:07.583647    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:18.820791 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:18.820697    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:18.882314 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:18.882279    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:18.932611 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:13:18.932577    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:13.819110 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:13.819076    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:13.904939 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:13.904895    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:15.552384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:15.552337    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:15.613304 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:15.613276    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:25.768281 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:25.768245    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:25.855467 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:25.855428    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.065711 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:26.065627    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.166508 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:26.166478    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:26.226145 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:26.226117    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:29.146501 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:29.146468    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:29.247399 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:29.247370    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:29.348647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:14:29.348620    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:19.838884 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:19.838855    2552 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:15:24.987123 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:24.987089    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:26.728312 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:26.728278    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.154509 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:30.154476    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.363323 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:30.363293    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.597789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:30.597752    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.698087 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:30.698057    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:30.757969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:30.757934    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:53.373426 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:53.370091    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:53.454076 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:15:53.454044    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:00.336780 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:00.336746    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:00.607768 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:00.607684    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:00.778483 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:00.778444    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:00.858924 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:00.858843    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.131752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.131656    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.296623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.296585    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.394367 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.394291    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.494203 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.494172    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.691471 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.691381    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.769953 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.769920    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.855483 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.855458    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:01.954560 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:01.954534    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:02.055630 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:02.055602    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:26.186916 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:16:26.186824    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.724467 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:23.724434    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.190585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.190505    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.276497 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.276458    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.284808 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.284776    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:17:24.288257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.288237    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.292347 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.292328    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-55ntd\""
Jul 28 17:17:24.292437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.292356    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:17:24.292437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.292366    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:17:24.292437 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.292361    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:17:24.302674 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.302656    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:17:24.320678 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.320658    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-pv5cz"]
Jul 28 17:17:24.323696 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.323682    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.324165 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.324148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.324252 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.324174    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdpf\" (UniqueName: \"kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.326425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.326406    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:17:24.326551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.326538    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lcbwl\""
Jul 28 17:17:24.335329 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.335309    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pv5cz"]
Jul 28 17:17:24.348896 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.348868    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.424900 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.424864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhbc\" (UniqueName: \"kubernetes.io/projected/f986c8a2-276f-4ced-971f-fcbc5968d76a-kube-api-access-kxhbc\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.425050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.424911    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f986c8a2-276f-4ced-971f-fcbc5968d76a-data\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.425050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.424938    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.425050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.424963    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdpf\" (UniqueName: \"kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.427222 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.427203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.437354 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.437330    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdpf\" (UniqueName: \"kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf\") pod \"kserve-controller-manager-cd4d44c67-zvpgh\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") " pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.445807 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.445756    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.526107 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.526078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f986c8a2-276f-4ced-971f-fcbc5968d76a-data\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.526205 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.526176    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhbc\" (UniqueName: \"kubernetes.io/projected/f986c8a2-276f-4ced-971f-fcbc5968d76a-kube-api-access-kxhbc\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.526481 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.526466    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f986c8a2-276f-4ced-971f-fcbc5968d76a-data\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.535037 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.535014    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhbc\" (UniqueName: \"kubernetes.io/projected/f986c8a2-276f-4ced-971f-fcbc5968d76a-kube-api-access-kxhbc\") pod \"seaweedfs-64568bcd49-pv5cz\" (UID: \"f986c8a2-276f-4ced-971f-fcbc5968d76a\") " pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.599086 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.599062    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:24.632220 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.632190    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:24.751417 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.751385    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:17:24.754557 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:17:24.754527    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053129d7_4fa6_4d25_a5b6_9195b3ff1b97.slice/crio-e5a0af99b50af671949d882bfe3be8dc3dd26a4b66b7d340beac6386e6dbc9f1 WatchSource:0}: Error finding container e5a0af99b50af671949d882bfe3be8dc3dd26a4b66b7d340beac6386e6dbc9f1: Status 404 returned error can't find the container with id e5a0af99b50af671949d882bfe3be8dc3dd26a4b66b7d340beac6386e6dbc9f1
Jul 28 17:17:24.755719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.755704    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:17:24.775642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.775621    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-pv5cz"]
Jul 28 17:17:24.780128 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:17:24.780102    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf986c8a2_276f_4ced_971f_fcbc5968d76a.slice/crio-28946237d7a63edf7a8a70aa40017f0bed9d58a4b2150e12136ec3cdb7c3ee88 WatchSource:0}: Error finding container 28946237d7a63edf7a8a70aa40017f0bed9d58a4b2150e12136ec3cdb7c3ee88: Status 404 returned error can't find the container with id 28946237d7a63edf7a8a70aa40017f0bed9d58a4b2150e12136ec3cdb7c3ee88
Jul 28 17:17:24.940923 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:24.940887    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.045556 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:25.045490    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.114390 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:25.114364    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.299143 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:25.298981    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.572469 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:25.572348    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" event={"ID":"053129d7-4fa6-4d25-a5b6-9195b3ff1b97","Type":"ContainerStarted","Data":"e5a0af99b50af671949d882bfe3be8dc3dd26a4b66b7d340beac6386e6dbc9f1"}
Jul 28 17:17:25.574599 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:25.574562    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pv5cz" event={"ID":"f986c8a2-276f-4ced-971f-fcbc5968d76a","Type":"ContainerStarted","Data":"28946237d7a63edf7a8a70aa40017f0bed9d58a4b2150e12136ec3cdb7c3ee88"}
Jul 28 17:17:29.591291 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.591241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-pv5cz" event={"ID":"f986c8a2-276f-4ced-971f-fcbc5968d76a","Type":"ContainerStarted","Data":"f9b9abd072bc8f3aca231e58cbdcb290f7db7a3a102d3711227bf164a7b06081"}
Jul 28 17:17:29.591781 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.591312    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:17:29.592687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.592660    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" event={"ID":"053129d7-4fa6-4d25-a5b6-9195b3ff1b97","Type":"ContainerStarted","Data":"b99356814a77b4786f9266d7264b651e054534fc62662b6c16001797dec68a23"}
Jul 28 17:17:29.592786 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.592770    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:17:29.609029 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.608970    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-pv5cz" podStartSLOduration=1.788907328 podStartE2EDuration="5.60895773s" totalImagesPullingTime="3.820050402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:29.608387799 +0000 UTC m=+430.160697712" watchObservedRunningTime="2026-07-28 17:17:29.60895773 +0000 UTC m=+430.161267621"
Jul 28 17:17:29.626706 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:29.626670    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" podStartSLOduration=1.83037888 podStartE2EDuration="5.626661463s" totalImagesPullingTime="3.796282583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:29.62550884 +0000 UTC m=+430.177818730" watchObservedRunningTime="2026-07-28 17:17:29.626661463 +0000 UTC m=+430.178971354"
Jul 28 17:17:35.600683 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:17:35.600650    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-pv5cz"
Jul 28 17:18:00.603797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:00.603765    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:18:08.516688 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.516651    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:18:08.517201 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.516854    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" podUID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" containerName="manager" containerID="cri-o://b99356814a77b4786f9266d7264b651e054534fc62662b6c16001797dec68a23" gracePeriod=10
Jul 28 17:18:08.539569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.539546    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-kj262"]
Jul 28 17:18:08.541580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.541566    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.553328 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.553304    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-kj262"]
Jul 28 17:18:08.653228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.653202    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abee3f1-749d-427f-a164-88293c7580d9-cert\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.653388 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.653274    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtgjv\" (UniqueName: \"kubernetes.io/projected/2abee3f1-749d-427f-a164-88293c7580d9-kube-api-access-xtgjv\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.733361 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.733327    2552 generic.go:356] "Generic (PLEG): container finished" podID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" containerID="b99356814a77b4786f9266d7264b651e054534fc62662b6c16001797dec68a23" exitCode=0
Jul 28 17:18:08.733515 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.733400    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" event={"ID":"053129d7-4fa6-4d25-a5b6-9195b3ff1b97","Type":"ContainerDied","Data":"b99356814a77b4786f9266d7264b651e054534fc62662b6c16001797dec68a23"}
Jul 28 17:18:08.750168 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.750146    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:18:08.754178 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.754160    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abee3f1-749d-427f-a164-88293c7580d9-cert\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.754264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.754210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtgjv\" (UniqueName: \"kubernetes.io/projected/2abee3f1-749d-427f-a164-88293c7580d9-kube-api-access-xtgjv\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.756384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.756361    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2abee3f1-749d-427f-a164-88293c7580d9-cert\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.763642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.763622    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtgjv\" (UniqueName: \"kubernetes.io/projected/2abee3f1-749d-427f-a164-88293c7580d9-kube-api-access-xtgjv\") pod \"kserve-controller-manager-cd4d44c67-kj262\" (UID: \"2abee3f1-749d-427f-a164-88293c7580d9\") " pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.855345 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.855279    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvdpf\" (UniqueName: \"kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf\") pod \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") "
Jul 28 17:18:08.855345 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.855331    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert\") pod \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\" (UID: \"053129d7-4fa6-4d25-a5b6-9195b3ff1b97\") "
Jul 28 17:18:08.857376 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.857350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert" (OuterVolumeSpecName: "cert") pod "053129d7-4fa6-4d25-a5b6-9195b3ff1b97" (UID: "053129d7-4fa6-4d25-a5b6-9195b3ff1b97"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:08.857454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.857433    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf" (OuterVolumeSpecName: "kube-api-access-lvdpf") pod "053129d7-4fa6-4d25-a5b6-9195b3ff1b97" (UID: "053129d7-4fa6-4d25-a5b6-9195b3ff1b97"). InnerVolumeSpecName "kube-api-access-lvdpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:08.896597 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.896574    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:08.956552 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.956526    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdpf\" (UniqueName: \"kubernetes.io/projected/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-kube-api-access-lvdpf\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:18:08.956552 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:08.956550    2552 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053129d7-4fa6-4d25-a5b6-9195b3ff1b97-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:18:09.018863 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.018714    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-kj262"]
Jul 28 17:18:09.020961 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:18:09.020937    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abee3f1_749d_427f_a164_88293c7580d9.slice/crio-9c09521ad351268809ca7f5c60ed0b4193d9abd26fad15011baf3dff36077dea WatchSource:0}: Error finding container 9c09521ad351268809ca7f5c60ed0b4193d9abd26fad15011baf3dff36077dea: Status 404 returned error can't find the container with id 9c09521ad351268809ca7f5c60ed0b4193d9abd26fad15011baf3dff36077dea
Jul 28 17:18:09.738149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.738115    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-kj262" event={"ID":"2abee3f1-749d-427f-a164-88293c7580d9","Type":"ContainerStarted","Data":"a534724dc5e2eca5257aac7ac1c6b3d96ddbb8ae3fd379b50cd65181f27456d7"}
Jul 28 17:18:09.738539 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.738175    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:09.738539 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.738197    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-kj262" event={"ID":"2abee3f1-749d-427f-a164-88293c7580d9","Type":"ContainerStarted","Data":"9c09521ad351268809ca7f5c60ed0b4193d9abd26fad15011baf3dff36077dea"}
Jul 28 17:18:09.739392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.739376    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh"
Jul 28 17:18:09.739480 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.739379    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-zvpgh" event={"ID":"053129d7-4fa6-4d25-a5b6-9195b3ff1b97","Type":"ContainerDied","Data":"e5a0af99b50af671949d882bfe3be8dc3dd26a4b66b7d340beac6386e6dbc9f1"}
Jul 28 17:18:09.739524 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.739512    2552 scope.go:117] "RemoveContainer" containerID="b99356814a77b4786f9266d7264b651e054534fc62662b6c16001797dec68a23"
Jul 28 17:18:09.759590 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.757330    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-kj262" podStartSLOduration=1.360955761 podStartE2EDuration="1.757307339s" totalImagesPullingTime="396.351578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:09.753416448 +0000 UTC m=+470.305726350" watchObservedRunningTime="2026-07-28 17:18:09.757307339 +0000 UTC m=+470.309617231"
Jul 28 17:18:09.769425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.769399    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:18:09.773497 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.773475    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-zvpgh"]
Jul 28 17:18:09.970948 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:09.970916    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" path="/var/lib/kubelet/pods/053129d7-4fa6-4d25-a5b6-9195b3ff1b97/volumes"
Jul 28 17:18:27.090296 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:27.090260    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:27.158894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:27.158864    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.087113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:38.087078    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.142182 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:38.142153    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:38.193976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:38.193947    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:40.749149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:40.749122    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-kj262"
Jul 28 17:18:41.719896 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.719865    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:41.785268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.785232    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-c2v7k"]
Jul 28 17:18:41.785698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.785659    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" containerName="manager"
Jul 28 17:18:41.785698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.785677    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" containerName="manager"
Jul 28 17:18:41.785805 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.785754    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="053129d7-4fa6-4d25-a5b6-9195b3ff1b97" containerName="manager"
Jul 28 17:18:41.788493 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.788470    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:41.791510 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.791491    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:18:41.792370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.792351    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-9psng\""
Jul 28 17:18:41.796127 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.795827    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:41.801372 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.801348    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-c2v7k"]
Jul 28 17:18:41.802682 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.802662    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-rxtkg"]
Jul 28 17:18:41.805565 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.805551    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:41.808383 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.808364    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xcwxj\""
Jul 28 17:18:41.808715 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.808700    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:18:41.816389 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.816368    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-rxtkg"]
Jul 28 17:18:41.891618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.891590    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-tls-certs\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:41.891763 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.891624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdjz\" (UniqueName: \"kubernetes.io/projected/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-kube-api-access-6vdjz\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:41.951911 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.951881    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:41.992767 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.992704    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c2e47ce-75c3-469e-a024-1566394c40f8-cert\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:41.992767 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.992746    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-tls-certs\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:41.992767 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.992768    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdjz\" (UniqueName: \"kubernetes.io/projected/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-kube-api-access-6vdjz\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:41.993034 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.992808    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwt7\" (UniqueName: \"kubernetes.io/projected/2c2e47ce-75c3-469e-a024-1566394c40f8-kube-api-access-jkwt7\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:41.995040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:41.995019    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-tls-certs\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:42.002318 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.002297    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdjz\" (UniqueName: \"kubernetes.io/projected/533ea5cb-5ec5-45d5-9a90-9b2e5293903f-kube-api-access-6vdjz\") pod \"model-serving-api-7b954ffc95-c2v7k\" (UID: \"533ea5cb-5ec5-45d5-9a90-9b2e5293903f\") " pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:42.049935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.049910    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.094022 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.093974    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c2e47ce-75c3-469e-a024-1566394c40f8-cert\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:42.094114 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.094064    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwt7\" (UniqueName: \"kubernetes.io/projected/2c2e47ce-75c3-469e-a024-1566394c40f8-kube-api-access-jkwt7\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:42.096244 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.096225    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c2e47ce-75c3-469e-a024-1566394c40f8-cert\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:42.099084 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.099066    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:42.103877 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.103855    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwt7\" (UniqueName: \"kubernetes.io/projected/2c2e47ce-75c3-469e-a024-1566394c40f8-kube-api-access-jkwt7\") pod \"odh-model-controller-75d9cdbddf-rxtkg\" (UID: \"2c2e47ce-75c3-469e-a024-1566394c40f8\") " pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:42.116837 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.116819    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:42.235734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.235702    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-c2v7k"]
Jul 28 17:18:42.238566 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:18:42.238536    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533ea5cb_5ec5_45d5_9a90_9b2e5293903f.slice/crio-1d217290acb64c5e9deb1f58e41367af210ac4de6863afa2962d2e284c0e52ab WatchSource:0}: Error finding container 1d217290acb64c5e9deb1f58e41367af210ac4de6863afa2962d2e284c0e52ab: Status 404 returned error can't find the container with id 1d217290acb64c5e9deb1f58e41367af210ac4de6863afa2962d2e284c0e52ab
Jul 28 17:18:42.256095 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.256076    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-rxtkg"]
Jul 28 17:18:42.257869 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:18:42.257846    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e47ce_75c3_469e_a024_1566394c40f8.slice/crio-7f701514141cb1f6237abcf9a572a13f8ce2ae2d8bbee2adab40dacc1a423503 WatchSource:0}: Error finding container 7f701514141cb1f6237abcf9a572a13f8ce2ae2d8bbee2adab40dacc1a423503: Status 404 returned error can't find the container with id 7f701514141cb1f6237abcf9a572a13f8ce2ae2d8bbee2adab40dacc1a423503
Jul 28 17:18:42.574013 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.573908    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.641677 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.641641    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:42.868215 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.868134    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg" event={"ID":"2c2e47ce-75c3-469e-a024-1566394c40f8","Type":"ContainerStarted","Data":"7f701514141cb1f6237abcf9a572a13f8ce2ae2d8bbee2adab40dacc1a423503"}
Jul 28 17:18:42.869906 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:42.869875    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-c2v7k" event={"ID":"533ea5cb-5ec5-45d5-9a90-9b2e5293903f","Type":"ContainerStarted","Data":"1d217290acb64c5e9deb1f58e41367af210ac4de6863afa2962d2e284c0e52ab"}
Jul 28 17:18:46.889375 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.889336    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg" event={"ID":"2c2e47ce-75c3-469e-a024-1566394c40f8","Type":"ContainerStarted","Data":"117249e8904c053e85ab7f2feb020eb66e90ffe1ba5e398b76fe4957332f3c3a"}
Jul 28 17:18:46.889820 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.889504    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:46.890861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.890839    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-c2v7k" event={"ID":"533ea5cb-5ec5-45d5-9a90-9b2e5293903f","Type":"ContainerStarted","Data":"c4e843af4c791c3695263ee32a98f4d2bb5655d89d59865806ab4764c42a0af6"}
Jul 28 17:18:46.891002 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.890954    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:18:46.906834 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.906787    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg" podStartSLOduration=2.311079392 podStartE2EDuration="5.906776518s" totalImagesPullingTime="3.595697126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:46.905673432 +0000 UTC m=+507.457983323" watchObservedRunningTime="2026-07-28 17:18:46.906776518 +0000 UTC m=+507.459086408"
Jul 28 17:18:46.925345 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:46.925302    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-c2v7k" podStartSLOduration=2.355857602 podStartE2EDuration="5.925288518s" totalImagesPullingTime="3.569430916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:46.924054621 +0000 UTC m=+507.476364541" watchObservedRunningTime="2026-07-28 17:18:46.925288518 +0000 UTC m=+507.477598409"
Jul 28 17:18:57.897605 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:57.897576    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-rxtkg"
Jul 28 17:18:57.899360 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:18:57.899339    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-c2v7k"
Jul 28 17:19:01.228454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:19:01.228421    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:01.282154 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:19:01.282129    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:08.548564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:19:08.548526    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:28.461779 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:19:28.461698    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:24.803844 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:24.803800    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:24.903969 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:24.903938    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:36.367102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.367066    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:36.591417 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.591385    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:20:36.594563 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.594546    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.597444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.597419    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d0cd1-kube-rbac-proxy-sar-config\""
Jul 28 17:20:36.597578 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.597471    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:20:36.598669 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.598640    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-wxvxt\""
Jul 28 17:20:36.598800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.598720    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:20:36.598800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.598729    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d0cd1-predictor-serving-cert\""
Jul 28 17:20:36.603934 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.603889    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:20:36.635481 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.635425    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.635481 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.635467    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tbz\" (UniqueName: \"kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.635614 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.635537    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.736139 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.736107    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.736299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.736168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.736299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.736204    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tbz\" (UniqueName: \"kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.736844 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.736826    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.738607 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.738579    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.745049 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.745028    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tbz\" (UniqueName: \"kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz\") pod \"success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") " pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.869487 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.869458    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:20:36.872875 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.872856    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:36.875450 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.875433    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d0cd1-predictor-serving-cert\""
Jul 28 17:20:36.875737 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.875724    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d0cd1-kube-rbac-proxy-sar-config\""
Jul 28 17:20:36.883307 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.883286    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:20:36.905882 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.905810    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:36.937931 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.937900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:36.938081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.937947    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:36.938081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.937980    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6dv\" (UniqueName: \"kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:36.982234 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.982202    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:20:36.986196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.986175    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:36.989018 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.988959    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:20:36.989142 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.989094    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:20:36.989142 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.989117    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 28 17:20:36.989259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.989121    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 28 17:20:36.999505 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:36.999483    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:20:37.038780 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038744    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.038926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.038926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038830    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.038926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038860    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.038926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038877    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6dv\" (UniqueName: \"kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.038926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.039229 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.038927    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.039229 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.039022    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwcg\" (UniqueName: \"kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.039229 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.039030    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-serving-cert: secret "error-404-isvc-d0cd1-predictor-serving-cert" not found
Jul 28 17:20:37.039229 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.039164    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls podName:53910743-7650-4424-8664-eacac1167e83 nodeName:}" failed. No retries permitted until 2026-07-28 17:20:37.53914193 +0000 UTC m=+618.091451816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls") pod "error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" (UID: "53910743-7650-4424-8664-eacac1167e83") : secret "error-404-isvc-d0cd1-predictor-serving-cert" not found
Jul 28 17:20:37.039530 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.039512    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.042361 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.042319    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:20:37.049782 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.049749    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6dv\" (UniqueName: \"kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.139521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139492    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.139657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.139657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139566    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwcg\" (UniqueName: \"kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.139657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139586    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.139657 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139609    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.139866 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.139733    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 17:20:37.139866 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.139801    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls podName:10241291-86e0-4bb5-93b5-008f308661e2 nodeName:}" failed. No retries permitted until 2026-07-28 17:20:37.639780981 +0000 UTC m=+618.192090867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls") pod "isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" (UID: "10241291-86e0-4bb5-93b5-008f308661e2") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 17:20:37.140021 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.139877    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.140260 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.140239    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.140357 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.140334    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.149036 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.149013    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwcg\" (UniqueName: \"kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.286899 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.286863    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerStarted","Data":"219ac8d5fe081c5f9de305389285065dd4106cfa48ab85eee1e4e1eabd2f48e2"}
Jul 28 17:20:37.543044 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.542962    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.545237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.545218    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") pod \"error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") " pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.643698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.643658    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:37.643878 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.643855    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 17:20:37.643949 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:20:37.643933    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls podName:10241291-86e0-4bb5-93b5-008f308661e2 nodeName:}" failed. No retries permitted until 2026-07-28 17:20:38.643913694 +0000 UTC m=+619.196223565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls") pod "isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" (UID: "10241291-86e0-4bb5-93b5-008f308661e2") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found
Jul 28 17:20:37.786493 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.785608    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:37.786493 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.786371    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:37.888458 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.885408    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:37.980679 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:37.980595    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:20:37.986964 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:20:37.986934    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53910743_7650_4424_8664_eacac1167e83.slice/crio-6d2df68fa280c332f5a1c78ead65ad3c30b606dd69b9bc41f4e5ee487a1bc718 WatchSource:0}: Error finding container 6d2df68fa280c332f5a1c78ead65ad3c30b606dd69b9bc41f4e5ee487a1bc718: Status 404 returned error can't find the container with id 6d2df68fa280c332f5a1c78ead65ad3c30b606dd69b9bc41f4e5ee487a1bc718
Jul 28 17:20:38.294459 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:38.294421    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerStarted","Data":"6d2df68fa280c332f5a1c78ead65ad3c30b606dd69b9bc41f4e5ee487a1bc718"}
Jul 28 17:20:38.675074 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:38.674838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:38.682829 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:38.682745    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:38.799034 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:38.799002    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:20:39.033747 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:39.033015    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:20:39.309815 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:39.309749    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerStarted","Data":"df9e7ee3567c1dc21ae251bbfc89b859103001283ce46cb126dae2e189bb593b"}
Jul 28 17:20:55.402627 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:55.402584    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerStarted","Data":"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"}
Jul 28 17:20:55.405008 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:55.404962    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerStarted","Data":"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"}
Jul 28 17:20:55.406624 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:55.406596    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerStarted","Data":"20f8e406e77f1b8c9269fce704676bf75fa226335d650e6ff03839b8b992144e"}
Jul 28 17:20:57.416055 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.416021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerStarted","Data":"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"}
Jul 28 17:20:57.416448 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.416154    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:57.416448 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.416200    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:20:57.417726 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.417697    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:20:57.418155 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.418135    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerStarted","Data":"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"}
Jul 28 17:20:57.418258 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.418247    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:57.418302 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.418273    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:20:57.419277 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.419256    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:20:57.436599 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.436561    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podStartSLOduration=2.204320136 podStartE2EDuration="21.43654905s" totalImagesPullingTime="19.232228914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:57.434737108 +0000 UTC m=+637.987047010" watchObservedRunningTime="2026-07-28 17:20:57.43654905 +0000 UTC m=+637.988858940"
Jul 28 17:20:57.454770 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:57.454711    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podStartSLOduration=1.272724024 podStartE2EDuration="21.454693054s" totalImagesPullingTime="20.18196903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:57.454298401 +0000 UTC m=+638.006608292" watchObservedRunningTime="2026-07-28 17:20:57.454693054 +0000 UTC m=+638.007002945"
Jul 28 17:20:58.401405 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.401373    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:58.424156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.424126    2552 generic.go:356] "Generic (PLEG): container finished" podID="10241291-86e0-4bb5-93b5-008f308661e2" containerID="20f8e406e77f1b8c9269fce704676bf75fa226335d650e6ff03839b8b992144e" exitCode=0
Jul 28 17:20:58.424468 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.424199    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerDied","Data":"20f8e406e77f1b8c9269fce704676bf75fa226335d650e6ff03839b8b992144e"}
Jul 28 17:20:58.424516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.424471    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:20:58.424570 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.424547    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:20:58.502211 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:20:58.502176    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:03.429844 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:03.429814    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:21:03.430323 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:03.430209    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:21:03.430371 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:03.430316    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:03.430565 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:03.430532    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:21:04.451491 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:04.451460    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerStarted","Data":"6c387d45a399decd011f73ac0190928aa854eab1a85b68092cc1507f1086c902"}
Jul 28 17:21:05.458299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:05.458266    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerStarted","Data":"9b15badf24631981f4dd228d32e5399ece0fd5f255afbada437f851da4f2a047"}
Jul 28 17:21:05.458678 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:05.458382    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:21:05.477899 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:05.477853    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podStartSLOduration=4.300032572 podStartE2EDuration="29.477839102s" totalImagesPullingTime="21.392549751s" totalInitContainerRuntime="3.785256779s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:05.47655231 +0000 UTC m=+646.028862213" watchObservedRunningTime="2026-07-28 17:21:05.477839102 +0000 UTC m=+646.030148991"
Jul 28 17:21:06.462389 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:06.462361    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:21:06.463637 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:06.463609    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:07.466810 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:07.466770    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:12.471187 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:12.471155    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:21:12.471690 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:12.471666    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:13.430764 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:13.430733    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:21:13.430918 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:13.430735    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:22.472509 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:22.472469    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:23.430767 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:23.430730    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:23.430931 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:23.430730    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:21:32.472577 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:32.472534    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:33.430697 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:33.430658    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:21:33.430888 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:33.430661    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:21:41.885871 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:41.885822    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:42.471645 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:42.471606    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:43.431212 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:43.431178    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:21:43.431592 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:43.431258    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:21:52.471794 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:52.471758    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:21:56.401070 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:56.401034    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:56.891760 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:21:56.891724    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:02.472136 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:02.472095    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:22:10.681334 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.681290    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:10.806484 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.806454    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:22:10.806694 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.806671    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" containerID="cri-o://012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d" gracePeriod=30
Jul 28 17:22:10.806770 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.806713    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kube-rbac-proxy" containerID="cri-o://854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c" gracePeriod=30
Jul 28 17:22:10.829196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.828982    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:10.861259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.861228    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:22:10.864569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.864547    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:10.867505 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.867485    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1fbfc-predictor-serving-cert\""
Jul 28 17:22:10.867614 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.867505    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1fbfc-kube-rbac-proxy-sar-config\""
Jul 28 17:22:10.875678 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.875653    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:22:10.907970 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.907941    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:10.927067 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.927042    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:22:10.927268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.927240    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" containerID="cri-o://7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0" gracePeriod=30
Jul 28 17:22:10.927358 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.927276    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kube-rbac-proxy" containerID="cri-o://cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3" gracePeriod=30
Jul 28 17:22:10.951176 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.951154    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:22:10.954339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.954319    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:10.957351 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.957191    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1fbfc-kube-rbac-proxy-sar-config\""
Jul 28 17:22:10.957351 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.957191    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1fbfc-predictor-serving-cert\""
Jul 28 17:22:10.957511 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.957477    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:10.957576 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.957558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:10.957643 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.957624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr8q\" (UniqueName: \"kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:10.969148 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:10.969127    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:22:11.008641 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.008610    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:11.058776 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.058746    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.058905 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.058798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.058951 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.058900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd52n\" (UniqueName: \"kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.058951 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.058929    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.058951 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.058948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.059120 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.059034    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr8q\" (UniqueName: \"kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.059483 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.059457    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.061372 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.061354    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.066772 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.066737    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr8q\" (UniqueName: \"kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q\") pod \"success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") " pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.159890 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.159858    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.160078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.159919    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd52n\" (UniqueName: \"kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.160078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.159936    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.161648 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.161168    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.166827 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.164434    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.168744 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.168724    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd52n\" (UniqueName: \"kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n\") pod \"error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") " pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.177013 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.176981    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.193271 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.193236    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:11.265654 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.265611    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.310410 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.310385    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:22:11.312597 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:22:11.312565    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c17a24_db85_4298_a8f4_762c1077759e.slice/crio-3868efd4d44b741de0f2d4846df849ac8717c6d1605021ad136d6edf27352de7 WatchSource:0}: Error finding container 3868efd4d44b741de0f2d4846df849ac8717c6d1605021ad136d6edf27352de7: Status 404 returned error can't find the container with id 3868efd4d44b741de0f2d4846df849ac8717c6d1605021ad136d6edf27352de7
Jul 28 17:22:11.406443 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.406421    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:22:11.407515 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:22:11.407483    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315213c7_2b98_4a29_9239_76401beb51f7.slice/crio-ce24128a134b8a421689929c7ade00561ef08d99e94b6d626938111b45cadd9a WatchSource:0}: Error finding container ce24128a134b8a421689929c7ade00561ef08d99e94b6d626938111b45cadd9a: Status 404 returned error can't find the container with id ce24128a134b8a421689929c7ade00561ef08d99e94b6d626938111b45cadd9a
Jul 28 17:22:11.723249 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.723170    2552 generic.go:356] "Generic (PLEG): container finished" podID="386bb619-5985-44dd-a988-1aefd39cdf65" containerID="854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c" exitCode=2
Jul 28 17:22:11.723249 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.723239    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerDied","Data":"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"}
Jul 28 17:22:11.724761 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.724741    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerStarted","Data":"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"}
Jul 28 17:22:11.724901 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.724782    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerStarted","Data":"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"}
Jul 28 17:22:11.724901 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.724817    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.724901 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.724844    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerStarted","Data":"3868efd4d44b741de0f2d4846df849ac8717c6d1605021ad136d6edf27352de7"}
Jul 28 17:22:11.724901 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.724870    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:11.726484 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726461    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:11.726847 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726821    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerStarted","Data":"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"}
Jul 28 17:22:11.726913 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726891    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.726980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726923    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerStarted","Data":"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"}
Jul 28 17:22:11.726980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726953    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerStarted","Data":"ce24128a134b8a421689929c7ade00561ef08d99e94b6d626938111b45cadd9a"}
Jul 28 17:22:11.727096 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.726985    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:11.727964 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.727929    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:11.728486 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.728471    2552 generic.go:356] "Generic (PLEG): container finished" podID="53910743-7650-4424-8664-eacac1167e83" containerID="cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3" exitCode=2
Jul 28 17:22:11.728590 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.728509    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerDied","Data":"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"}
Jul 28 17:22:11.745323 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.745283    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podStartSLOduration=1.745270241 podStartE2EDuration="1.745270241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:11.743026277 +0000 UTC m=+712.295336167" watchObservedRunningTime="2026-07-28 17:22:11.745270241 +0000 UTC m=+712.297580131"
Jul 28 17:22:11.763571 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:11.763524    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podStartSLOduration=1.7635084779999999 podStartE2EDuration="1.763508478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:11.762335999 +0000 UTC m=+712.314645929" watchObservedRunningTime="2026-07-28 17:22:11.763508478 +0000 UTC m=+712.315818369"
Jul 28 17:22:12.472034 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:12.471984    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 17:22:12.733966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:12.733874    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:12.733966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:12.733925    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:13.424854 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:13.424813    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 17:22:13.425016 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:13.424820    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 17:22:13.431238 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:13.431215    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 17:22:13.431338 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:13.431236    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 17:22:14.226218 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.226197    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:22:14.285273 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.285243    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config\") pod \"386bb619-5985-44dd-a988-1aefd39cdf65\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") "
Jul 28 17:22:14.285418 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.285281    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls\") pod \"386bb619-5985-44dd-a988-1aefd39cdf65\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") "
Jul 28 17:22:14.285418 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.285315    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79tbz\" (UniqueName: \"kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz\") pod \"386bb619-5985-44dd-a988-1aefd39cdf65\" (UID: \"386bb619-5985-44dd-a988-1aefd39cdf65\") "
Jul 28 17:22:14.285679 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.285643    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "386bb619-5985-44dd-a988-1aefd39cdf65" (UID: "386bb619-5985-44dd-a988-1aefd39cdf65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:14.287409 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.287379    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "386bb619-5985-44dd-a988-1aefd39cdf65" (UID: "386bb619-5985-44dd-a988-1aefd39cdf65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:22:14.287576 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.287551    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz" (OuterVolumeSpecName: "kube-api-access-79tbz") pod "386bb619-5985-44dd-a988-1aefd39cdf65" (UID: "386bb619-5985-44dd-a988-1aefd39cdf65"). InnerVolumeSpecName "kube-api-access-79tbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:22:14.363193 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.363169    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:22:14.386580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.386553    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/386bb619-5985-44dd-a988-1aefd39cdf65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.386580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.386577    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/386bb619-5985-44dd-a988-1aefd39cdf65-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.386787 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.386594    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79tbz\" (UniqueName: \"kubernetes.io/projected/386bb619-5985-44dd-a988-1aefd39cdf65-kube-api-access-79tbz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.487072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.486975    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d6dv\" (UniqueName: \"kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv\") pod \"53910743-7650-4424-8664-eacac1167e83\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") "
Jul 28 17:22:14.487072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.487036    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") pod \"53910743-7650-4424-8664-eacac1167e83\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") "
Jul 28 17:22:14.487279 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.487119    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config\") pod \"53910743-7650-4424-8664-eacac1167e83\" (UID: \"53910743-7650-4424-8664-eacac1167e83\") "
Jul 28 17:22:14.487413 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.487389    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "53910743-7650-4424-8664-eacac1167e83" (UID: "53910743-7650-4424-8664-eacac1167e83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:22:14.489073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.489053    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv" (OuterVolumeSpecName: "kube-api-access-6d6dv") pod "53910743-7650-4424-8664-eacac1167e83" (UID: "53910743-7650-4424-8664-eacac1167e83"). InnerVolumeSpecName "kube-api-access-6d6dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:22:14.489166 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.489145    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "53910743-7650-4424-8664-eacac1167e83" (UID: "53910743-7650-4424-8664-eacac1167e83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:22:14.588618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.588585    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/53910743-7650-4424-8664-eacac1167e83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.588618 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.588614    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6dv\" (UniqueName: \"kubernetes.io/projected/53910743-7650-4424-8664-eacac1167e83-kube-api-access-6d6dv\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.588784 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.588627    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/53910743-7650-4424-8664-eacac1167e83-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:22:14.742982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.742895    2552 generic.go:356] "Generic (PLEG): container finished" podID="53910743-7650-4424-8664-eacac1167e83" containerID="7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0" exitCode=0
Jul 28 17:22:14.742982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.742960    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"
Jul 28 17:22:14.742982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.742975    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerDied","Data":"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"}
Jul 28 17:22:14.743232 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.743051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv" event={"ID":"53910743-7650-4424-8664-eacac1167e83","Type":"ContainerDied","Data":"6d2df68fa280c332f5a1c78ead65ad3c30b606dd69b9bc41f4e5ee487a1bc718"}
Jul 28 17:22:14.743232 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.743084    2552 scope.go:117] "RemoveContainer" containerID="cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"
Jul 28 17:22:14.744588 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.744562    2552 generic.go:356] "Generic (PLEG): container finished" podID="386bb619-5985-44dd-a988-1aefd39cdf65" containerID="012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d" exitCode=0
Jul 28 17:22:14.744696 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.744610    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"
Jul 28 17:22:14.744696 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.744645    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerDied","Data":"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"}
Jul 28 17:22:14.744696 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.744692    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6" event={"ID":"386bb619-5985-44dd-a988-1aefd39cdf65","Type":"ContainerDied","Data":"219ac8d5fe081c5f9de305389285065dd4106cfa48ab85eee1e4e1eabd2f48e2"}
Jul 28 17:22:14.752264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.752241    2552 scope.go:117] "RemoveContainer" containerID="7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"
Jul 28 17:22:14.760185 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.760168    2552 scope.go:117] "RemoveContainer" containerID="cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"
Jul 28 17:22:14.760454 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:22:14.760438    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3\": container with ID starting with cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3 not found: ID does not exist" containerID="cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"
Jul 28 17:22:14.760501 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.760463    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3"} err="failed to get container status \"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3\": rpc error: code = NotFound desc = could not find container \"cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3\": container with ID starting with cbc2bb79c304f47b4b0b62637a30b092eb03a59d80d80a6fa531cecda43872a3 not found: ID does not exist"
Jul 28 17:22:14.760501 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.760479    2552 scope.go:117] "RemoveContainer" containerID="7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"
Jul 28 17:22:14.760715 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:22:14.760696    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0\": container with ID starting with 7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0 not found: ID does not exist" containerID="7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"
Jul 28 17:22:14.760752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.760723    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0"} err="failed to get container status \"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0\": rpc error: code = NotFound desc = could not find container \"7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0\": container with ID starting with 7259d61d8b43391d74593d8ca73fc598eef6b888dfabdcfe64b71da90e8bbbf0 not found: ID does not exist"
Jul 28 17:22:14.760752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.760740    2552 scope.go:117] "RemoveContainer" containerID="854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"
Jul 28 17:22:14.767374 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.767355    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:22:14.768033 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.768021    2552 scope.go:117] "RemoveContainer" containerID="012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"
Jul 28 17:22:14.771976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.771953    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d0cd1-predictor-6d8bd88894-j7lhv"]
Jul 28 17:22:14.775207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.775188    2552 scope.go:117] "RemoveContainer" containerID="854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"
Jul 28 17:22:14.775435 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:22:14.775419    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c\": container with ID starting with 854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c not found: ID does not exist" containerID="854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"
Jul 28 17:22:14.775500 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.775443    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c"} err="failed to get container status \"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c\": rpc error: code = NotFound desc = could not find container \"854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c\": container with ID starting with 854eda0533ab507ad4dc6186a1e85ee9a012fa8534fcc8a0a4bf73a9db1b3d0c not found: ID does not exist"
Jul 28 17:22:14.775500 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.775464    2552 scope.go:117] "RemoveContainer" containerID="012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"
Jul 28 17:22:14.775729 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:22:14.775710    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d\": container with ID starting with 012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d not found: ID does not exist" containerID="012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"
Jul 28 17:22:14.775768 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.775734    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d"} err="failed to get container status \"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d\": rpc error: code = NotFound desc = could not find container \"012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d\": container with ID starting with 012a92942cbf39c444261d2b973f7100c5bb7ebfd310949804141921de99b34d not found: ID does not exist"
Jul 28 17:22:14.788467 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.785715    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:22:14.790668 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:14.790648    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d0cd1-predictor-58d767b4ff-s94d6"]
Jul 28 17:22:15.972052 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:15.972019    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" path="/var/lib/kubelet/pods/386bb619-5985-44dd-a988-1aefd39cdf65/volumes"
Jul 28 17:22:15.972462 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:15.972448    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53910743-7650-4424-8664-eacac1167e83" path="/var/lib/kubelet/pods/53910743-7650-4424-8664-eacac1167e83/volumes"
Jul 28 17:22:17.738477 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:17.738450    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:17.738953 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:17.738934    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:22:17.739023 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:17.738974    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:17.739373 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:17.739344    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:22.472150 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:22.472121    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:22:27.739077 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:27.739039    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:27.744421 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:27.744360    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:37.739283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:37.739239    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:37.739651 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:37.739365    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:41.156606 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:41.156571    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:41.257634 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:41.257603    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:47.739687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:47.739646    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:22:47.740092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:47.739651    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:22:56.441344 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:56.441300    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:57.104597 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:57.104558    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:57.739705 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:57.739680    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:22:57.740142 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:22:57.740121    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:23:06.555446 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.555412    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:06.665069 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.665039    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:23:06.665334 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.665307    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container" containerID="cri-o://6c387d45a399decd011f73ac0190928aa854eab1a85b68092cc1507f1086c902" gracePeriod=30
Jul 28 17:23:06.665454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.665351    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kube-rbac-proxy" containerID="cri-o://9b15badf24631981f4dd228d32e5399ece0fd5f255afbada437f851da4f2a047" gracePeriod=30
Jul 28 17:23:06.682167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.682137    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:06.737975 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.737948    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:23:06.738303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738290    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container"
Jul 28 17:23:06.738303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738304    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738314    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738322    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738334    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738340    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738350    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container"
Jul 28 17:23:06.738381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738356    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container"
Jul 28 17:23:06.738582 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738405    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738582 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738412    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kube-rbac-proxy"
Jul 28 17:23:06.738582 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738419    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="53910743-7650-4424-8664-eacac1167e83" containerName="kserve-container"
Jul 28 17:23:06.738582 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.738426    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="386bb619-5985-44dd-a988-1aefd39cdf65" containerName="kserve-container"
Jul 28 17:23:06.741716 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.741693    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.744363 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.744337    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a5ae-predictor-serving-cert\""
Jul 28 17:23:06.744465 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.744418    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3a5ae-kube-rbac-proxy-sar-config\""
Jul 28 17:23:06.750793 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.750770    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:23:06.760623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.760599    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:06.812159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.812087    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.812159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.812148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8t7\" (UniqueName: \"kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.812384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.812252    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.833158 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.833116    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:06.840729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.840707    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:23:06.844352 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.844334    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:06.846882 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.846867    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a5ae-predictor-serving-cert\""
Jul 28 17:23:06.846980 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.846890    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3a5ae-kube-rbac-proxy-sar-config\""
Jul 28 17:23:06.853094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.853064    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:23:06.913337 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:06.913474 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913355    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:06.913474 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913394    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8t7\" (UniqueName: \"kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.913474 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913437    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.913580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjn76\" (UniqueName: \"kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:06.913580 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.913517    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.914223 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.914200    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.915834 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.915818    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.924406 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.924384    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8t7\" (UniqueName: \"kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7\") pod \"success-200-isvc-3a5ae-predictor-69d6694669-t9t26\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") " pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:06.930246 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.930220    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:06.964180 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.964153    2552 generic.go:356] "Generic (PLEG): container finished" podID="10241291-86e0-4bb5-93b5-008f308661e2" containerID="9b15badf24631981f4dd228d32e5399ece0fd5f255afbada437f851da4f2a047" exitCode=2
Jul 28 17:23:06.964327 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:06.964203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerDied","Data":"9b15badf24631981f4dd228d32e5399ece0fd5f255afbada437f851da4f2a047"}
Jul 28 17:23:07.014324 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.014298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjn76\" (UniqueName: \"kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.014440 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.014348    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.014440 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.014400    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.014560 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:23:07.014530    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-serving-cert: secret "error-404-isvc-3a5ae-predictor-serving-cert" not found
Jul 28 17:23:07.014613 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:23:07.014605    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls podName:77284c84-3f73-41eb-9ab2-352a44895984 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:07.514589413 +0000 UTC m=+768.066899281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls") pod "error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" (UID: "77284c84-3f73-41eb-9ab2-352a44895984") : secret "error-404-isvc-3a5ae-predictor-serving-cert" not found
Jul 28 17:23:07.015042 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.015026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.023629 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.023606    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjn76\" (UniqueName: \"kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.052359 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.052339    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:07.186584 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.186559    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:23:07.188453 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:23:07.188430    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd61427_c7ab_4d72_aa02_a8018c09322b.slice/crio-5bd4f77d0cb3713387840d87c075d349bb0c3124a5a7f697281aad8d7db5fac3 WatchSource:0}: Error finding container 5bd4f77d0cb3713387840d87c075d349bb0c3124a5a7f697281aad8d7db5fac3: Status 404 returned error can't find the container with id 5bd4f77d0cb3713387840d87c075d349bb0c3124a5a7f697281aad8d7db5fac3
Jul 28 17:23:07.284854 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.284816    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:07.467155 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.467119    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 17:23:07.518392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.518366    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.520679 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.520649    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") pod \"error-404-isvc-3a5ae-predictor-7888f995f6-fzw55\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") " pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.755311 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.755236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:07.884868 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.884844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:23:07.972857 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.972829    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:07.973923 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.973900    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerStarted","Data":"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"}
Jul 28 17:23:07.973979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.973952    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:07.974031 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.973979    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:07.974031 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.974015    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerStarted","Data":"d24cd3801274db637503cbe761de6ba62a18a7c2aece9675328fdc3cc2f406eb"}
Jul 28 17:23:07.974094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.974036    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerStarted","Data":"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"}
Jul 28 17:23:07.974094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.974053    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerStarted","Data":"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"}
Jul 28 17:23:07.974094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.974069    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerStarted","Data":"5bd4f77d0cb3713387840d87c075d349bb0c3124a5a7f697281aad8d7db5fac3"}
Jul 28 17:23:07.995795 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:07.995753    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podStartSLOduration=1.995738377 podStartE2EDuration="1.995738377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:07.993931964 +0000 UTC m=+768.546241855" watchObservedRunningTime="2026-07-28 17:23:07.995738377 +0000 UTC m=+768.548048269"
Jul 28 17:23:08.976681 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.976644    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerStarted","Data":"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"}
Jul 28 17:23:08.977158 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.976819    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:08.977158 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.976856    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:08.977158 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.977132    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:08.978054 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.978031    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:08.996391 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:08.996337    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podStartSLOduration=2.996323296 podStartE2EDuration="2.996323296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:08.996105718 +0000 UTC m=+769.548415638" watchObservedRunningTime="2026-07-28 17:23:08.996323296 +0000 UTC m=+769.548633187"
Jul 28 17:23:09.980209 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:09.980164    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:09.980604 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:09.980244    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:10.912228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:10.912119    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:10.986314 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:10.986284    2552 generic.go:356] "Generic (PLEG): container finished" podID="10241291-86e0-4bb5-93b5-008f308661e2" containerID="6c387d45a399decd011f73ac0190928aa854eab1a85b68092cc1507f1086c902" exitCode=0
Jul 28 17:23:10.986615 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:10.986323    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerDied","Data":"6c387d45a399decd011f73ac0190928aa854eab1a85b68092cc1507f1086c902"}
Jul 28 17:23:11.163454 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.163368    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:11.407623 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.407600    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:23:11.452262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452180    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location\") pod \"10241291-86e0-4bb5-93b5-008f308661e2\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") "
Jul 28 17:23:11.452262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452223    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert\") pod \"10241291-86e0-4bb5-93b5-008f308661e2\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") "
Jul 28 17:23:11.452262 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452254    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkwcg\" (UniqueName: \"kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg\") pod \"10241291-86e0-4bb5-93b5-008f308661e2\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") "
Jul 28 17:23:11.452514 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452343    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config\") pod \"10241291-86e0-4bb5-93b5-008f308661e2\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") "
Jul 28 17:23:11.452514 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452387    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") pod \"10241291-86e0-4bb5-93b5-008f308661e2\" (UID: \"10241291-86e0-4bb5-93b5-008f308661e2\") "
Jul 28 17:23:11.452619 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452572    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10241291-86e0-4bb5-93b5-008f308661e2" (UID: "10241291-86e0-4bb5-93b5-008f308661e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:23:11.452677 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452623    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10241291-86e0-4bb5-93b5-008f308661e2" (UID: "10241291-86e0-4bb5-93b5-008f308661e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:11.452677 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452663    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10241291-86e0-4bb5-93b5-008f308661e2" (UID: "10241291-86e0-4bb5-93b5-008f308661e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:11.452780 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452756    2552 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10241291-86e0-4bb5-93b5-008f308661e2-kserve-provision-location\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:23:11.452813 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452792    2552 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-cabundle-cert\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:23:11.452852 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.452811    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10241291-86e0-4bb5-93b5-008f308661e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:23:11.454402 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.454373    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10241291-86e0-4bb5-93b5-008f308661e2" (UID: "10241291-86e0-4bb5-93b5-008f308661e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:11.454857 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.454841    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg" (OuterVolumeSpecName: "kube-api-access-pkwcg") pod "10241291-86e0-4bb5-93b5-008f308661e2" (UID: "10241291-86e0-4bb5-93b5-008f308661e2"). InnerVolumeSpecName "kube-api-access-pkwcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:23:11.553284 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.553249    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10241291-86e0-4bb5-93b5-008f308661e2-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:23:11.553284 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.553286    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwcg\" (UniqueName: \"kubernetes.io/projected/10241291-86e0-4bb5-93b5-008f308661e2-kube-api-access-pkwcg\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:23:11.991316 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.991284    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7" event={"ID":"10241291-86e0-4bb5-93b5-008f308661e2","Type":"ContainerDied","Data":"df9e7ee3567c1dc21ae251bbfc89b859103001283ce46cb126dae2e189bb593b"}
Jul 28 17:23:11.991316 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.991313    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"
Jul 28 17:23:11.991661 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.991361    2552 scope.go:117] "RemoveContainer" containerID="9b15badf24631981f4dd228d32e5399ece0fd5f255afbada437f851da4f2a047"
Jul 28 17:23:11.999015 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:11.998982    2552 scope.go:117] "RemoveContainer" containerID="6c387d45a399decd011f73ac0190928aa854eab1a85b68092cc1507f1086c902"
Jul 28 17:23:12.005926 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:12.005903    2552 scope.go:117] "RemoveContainer" containerID="20f8e406e77f1b8c9269fce704676bf75fa226335d650e6ff03839b8b992144e"
Jul 28 17:23:12.011204 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:12.011184    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:23:12.014824 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:12.014804    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-5bc9545b47-jvsx7"]
Jul 28 17:23:13.972312 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:13.972279    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10241291-86e0-4bb5-93b5-008f308661e2" path="/var/lib/kubelet/pods/10241291-86e0-4bb5-93b5-008f308661e2/volumes"
Jul 28 17:23:14.986051 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:14.986026    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:14.986416 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:14.986339    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:23:14.986458 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:14.986428    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:14.986733 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:14.986709    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:24.987369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:24.987332    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:24.987717 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:24.987332    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:34.986662 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:34.986626    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:34.987055 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:34.986772    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:44.987243 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:44.987166    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 17:23:44.987243 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:44.987165    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 17:23:54.987913 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:54.987879    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:23:54.988338 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:23:54.987971    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:24:06.820399 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:24:06.820357    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:07.215851 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:24:07.215820    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:25:41.141395 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:25:41.141359    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:15.005755 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:27:15.005722    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:15.066769 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:27:15.066737    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:25.184523 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:27:25.184488    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:25.241399 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:27:25.241373    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:12.934582 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:28:12.934499    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:19.607510 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:28:19.607482    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:25.671628 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.671553    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:25.770811 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.770777    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:31:25.771065 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.771043    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" containerID="cri-o://f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4" gracePeriod=30
Jul 28 17:31:25.771159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.771131    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kube-rbac-proxy" containerID="cri-o://333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681" gracePeriod=30
Jul 28 17:31:25.904425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904397    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:31:25.904766 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904753    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="storage-initializer"
Jul 28 17:31:25.904831 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904768    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="storage-initializer"
Jul 28 17:31:25.904831 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904789    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container"
Jul 28 17:31:25.904831 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904795    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container"
Jul 28 17:31:25.904831 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904811    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kube-rbac-proxy"
Jul 28 17:31:25.904831 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904817    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kube-rbac-proxy"
Jul 28 17:31:25.905126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904874    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kserve-container"
Jul 28 17:31:25.905126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.904882    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10241291-86e0-4bb5-93b5-008f308661e2" containerName="kube-rbac-proxy"
Jul 28 17:31:25.908316 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.908297    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:25.910915 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.910891    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b0164-predictor-serving-cert\""
Jul 28 17:31:25.911041 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.910985    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b0164-kube-rbac-proxy-sar-config\""
Jul 28 17:31:25.918044 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.917934    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:31:25.918812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.918230    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" containerID="cri-o://97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd" gracePeriod=30
Jul 28 17:31:25.918812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.918415    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kube-rbac-proxy" containerID="cri-o://931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b" gracePeriod=30
Jul 28 17:31:25.921603 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.921559    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:31:25.929791 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.929770    2552 generic.go:356] "Generic (PLEG): container finished" podID="73c17a24-db85-4298-a8f4-762c1077759e" containerID="333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681" exitCode=2
Jul 28 17:31:25.929886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.929847    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerDied","Data":"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"}
Jul 28 17:31:25.972202 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.972166    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:31:25.975516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.975494    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:25.978223 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.978204    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b0164-predictor-serving-cert\""
Jul 28 17:31:25.978325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.978256    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b0164-kube-rbac-proxy-sar-config\""
Jul 28 17:31:25.986745 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:25.986725    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:31:26.066317 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066289    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.066485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066345    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmtx\" (UniqueName: \"kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.066485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066399    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nd27\" (UniqueName: \"kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.066485 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066463    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.066655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066516    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.066655 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.066559    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.167845 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.167815    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.168011 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.167855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.168011 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.167880    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.168011 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.167910    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.168011 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.167973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmtx\" (UniqueName: \"kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.168246 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.168042    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nd27\" (UniqueName: \"kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.168648 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.168621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.168768 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.168622    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.170344 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.170326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.170434 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.170352    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.176274 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.176227    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmtx\" (UniqueName: \"kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx\") pod \"success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") " pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.176396 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.176379    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nd27\" (UniqueName: \"kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27\") pod \"error-404-isvc-b0164-predictor-dd9648568-nwtkk\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") " pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.221389 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.221367    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.285806 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.285760    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.357196 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.357171    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:31:26.360111 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:31:26.360082    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2a77b3_e9a3_4590_af3e_4f5477b72454.slice/crio-0989dd4f6d91a006c704de00825138cf853476f95a1c8fec06a8d635870a70b8 WatchSource:0}: Error finding container 0989dd4f6d91a006c704de00825138cf853476f95a1c8fec06a8d635870a70b8: Status 404 returned error can't find the container with id 0989dd4f6d91a006c704de00825138cf853476f95a1c8fec06a8d635870a70b8
Jul 28 17:31:26.422108 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.422084    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:31:26.424091 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:31:26.424066    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d8f25f_49af_4e5c_96cf_fef30bc44530.slice/crio-0f90701a0d2da3c6441e7b97c388a0df93f4e37e4c23fa375e9027fe05092ee5 WatchSource:0}: Error finding container 0f90701a0d2da3c6441e7b97c388a0df93f4e37e4c23fa375e9027fe05092ee5: Status 404 returned error can't find the container with id 0f90701a0d2da3c6441e7b97c388a0df93f4e37e4c23fa375e9027fe05092ee5
Jul 28 17:31:26.935649 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.935600    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerStarted","Data":"ab54b42ce625aec9953d57ce54b95e3aa5e0c24d7d8b7eefe0e58d888ee1cc40"}
Jul 28 17:31:26.936109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.935680    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.936109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.935711    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerStarted","Data":"fa17883b6df4928719303fd3b25e400da73e03bbf04c0fb0faa971dc80db33ca"}
Jul 28 17:31:26.936109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.935740    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:26.936109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.935768    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerStarted","Data":"0f90701a0d2da3c6441e7b97c388a0df93f4e37e4c23fa375e9027fe05092ee5"}
Jul 28 17:31:26.937157 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.937127    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:31:26.937441 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.937420    2552 generic.go:356] "Generic (PLEG): container finished" podID="315213c7-2b98-4a29-9239-76401beb51f7" containerID="931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b" exitCode=2
Jul 28 17:31:26.937489 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.937447    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerDied","Data":"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"}
Jul 28 17:31:26.938975 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.938956    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerStarted","Data":"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"}
Jul 28 17:31:26.939089 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.939045    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerStarted","Data":"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"}
Jul 28 17:31:26.939089 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.939070    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerStarted","Data":"0989dd4f6d91a006c704de00825138cf853476f95a1c8fec06a8d635870a70b8"}
Jul 28 17:31:26.939177 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.939095    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:26.954860 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.954819    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podStartSLOduration=1.954804893 podStartE2EDuration="1.954804893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:26.952961488 +0000 UTC m=+1267.505271381" watchObservedRunningTime="2026-07-28 17:31:26.954804893 +0000 UTC m=+1267.507114784"
Jul 28 17:31:26.970503 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:26.970462    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podStartSLOduration=1.970449372 podStartE2EDuration="1.970449372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:26.968654602 +0000 UTC m=+1267.520964493" watchObservedRunningTime="2026-07-28 17:31:26.970449372 +0000 UTC m=+1267.522759263"
Jul 28 17:31:27.052161 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.052128    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:31:27.734295 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.734248    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 17:31:27.734434 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.734259    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 17:31:27.739263 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.739238    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 17:31:27.739410 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.739392    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 17:31:27.943470 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.943439    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:27.943883 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.943567    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:31:27.944621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:27.944595    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:28.949446 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:28.949395    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:29.625284 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.625257    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:31:29.801556 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.801528    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls\") pod \"73c17a24-db85-4298-a8f4-762c1077759e\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") "
Jul 28 17:31:29.801704 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.801567    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config\") pod \"73c17a24-db85-4298-a8f4-762c1077759e\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") "
Jul 28 17:31:29.801704 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.801631    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr8q\" (UniqueName: \"kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q\") pod \"73c17a24-db85-4298-a8f4-762c1077759e\" (UID: \"73c17a24-db85-4298-a8f4-762c1077759e\") "
Jul 28 17:31:29.802003 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.801944    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73c17a24-db85-4298-a8f4-762c1077759e" (UID: "73c17a24-db85-4298-a8f4-762c1077759e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:31:29.804243 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.804218    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q" (OuterVolumeSpecName: "kube-api-access-vsr8q") pod "73c17a24-db85-4298-a8f4-762c1077759e" (UID: "73c17a24-db85-4298-a8f4-762c1077759e"). InnerVolumeSpecName "kube-api-access-vsr8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:29.804326 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.804272    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73c17a24-db85-4298-a8f4-762c1077759e" (UID: "73c17a24-db85-4298-a8f4-762c1077759e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:29.862457 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.862432    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:31:29.902729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.902708    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73c17a24-db85-4298-a8f4-762c1077759e-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:29.902729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.902729    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73c17a24-db85-4298-a8f4-762c1077759e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:29.902873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.902738    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr8q\" (UniqueName: \"kubernetes.io/projected/73c17a24-db85-4298-a8f4-762c1077759e-kube-api-access-vsr8q\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:29.956581 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.956549    2552 generic.go:356] "Generic (PLEG): container finished" podID="73c17a24-db85-4298-a8f4-762c1077759e" containerID="f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4" exitCode=0
Jul 28 17:31:29.956985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.956602    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"
Jul 28 17:31:29.956985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.956638    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerDied","Data":"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"}
Jul 28 17:31:29.956985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.956754    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x" event={"ID":"73c17a24-db85-4298-a8f4-762c1077759e","Type":"ContainerDied","Data":"3868efd4d44b741de0f2d4846df849ac8717c6d1605021ad136d6edf27352de7"}
Jul 28 17:31:29.956985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.956794    2552 scope.go:117] "RemoveContainer" containerID="333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"
Jul 28 17:31:29.958276 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.958236    2552 generic.go:356] "Generic (PLEG): container finished" podID="315213c7-2b98-4a29-9239-76401beb51f7" containerID="97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd" exitCode=0
Jul 28 17:31:29.958407 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.958279    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerDied","Data":"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"}
Jul 28 17:31:29.958407 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.958298    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"
Jul 28 17:31:29.958407 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.958319    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp" event={"ID":"315213c7-2b98-4a29-9239-76401beb51f7","Type":"ContainerDied","Data":"ce24128a134b8a421689929c7ade00561ef08d99e94b6d626938111b45cadd9a"}
Jul 28 17:31:29.965145 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.965128    2552 scope.go:117] "RemoveContainer" containerID="f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"
Jul 28 17:31:29.973153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.973134    2552 scope.go:117] "RemoveContainer" containerID="333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"
Jul 28 17:31:29.973386 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:31:29.973368    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681\": container with ID starting with 333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681 not found: ID does not exist" containerID="333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"
Jul 28 17:31:29.973443 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.973395    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681"} err="failed to get container status \"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681\": rpc error: code = NotFound desc = could not find container \"333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681\": container with ID starting with 333ac4aac2ec6a8453dec8bf5dc7e108bfc67c2fb21f0fda6168fa24f6186681 not found: ID does not exist"
Jul 28 17:31:29.973443 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.973412    2552 scope.go:117] "RemoveContainer" containerID="f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"
Jul 28 17:31:29.973665 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:31:29.973648    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4\": container with ID starting with f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4 not found: ID does not exist" containerID="f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"
Jul 28 17:31:29.973729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.973674    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4"} err="failed to get container status \"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4\": rpc error: code = NotFound desc = could not find container \"f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4\": container with ID starting with f4a25ee2901a5482428cd66a62db84821f7f0d0209d78b811189783f8ed8e2b4 not found: ID does not exist"
Jul 28 17:31:29.973729 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.973696    2552 scope.go:117] "RemoveContainer" containerID="931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"
Jul 28 17:31:29.980596 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.980554    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:31:29.980661 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.980605    2552 scope.go:117] "RemoveContainer" containerID="97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"
Jul 28 17:31:29.986391 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.986364    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1fbfc-predictor-6c545c88dc-pt88x"]
Jul 28 17:31:29.988747 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.988733    2552 scope.go:117] "RemoveContainer" containerID="931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"
Jul 28 17:31:29.988984 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:31:29.988965    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b\": container with ID starting with 931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b not found: ID does not exist" containerID="931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"
Jul 28 17:31:29.989061 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.989007    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b"} err="failed to get container status \"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b\": rpc error: code = NotFound desc = could not find container \"931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b\": container with ID starting with 931b4e5f3d632761099033c1c127c68a792ad5b28f4892cc8d2061b0a5bbe80b not found: ID does not exist"
Jul 28 17:31:29.989061 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.989028    2552 scope.go:117] "RemoveContainer" containerID="97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"
Jul 28 17:31:29.989273 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:31:29.989254    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd\": container with ID starting with 97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd not found: ID does not exist" containerID="97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"
Jul 28 17:31:29.989335 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:29.989278    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd"} err="failed to get container status \"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd\": rpc error: code = NotFound desc = could not find container \"97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd\": container with ID starting with 97e1678bc0f7e844706bcbe9caffa9182ffb4066fa7ffd15f3909bec1d2a0ebd not found: ID does not exist"
Jul 28 17:31:30.003549 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.003530    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config\") pod \"315213c7-2b98-4a29-9239-76401beb51f7\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") "
Jul 28 17:31:30.003622 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.003583    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd52n\" (UniqueName: \"kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n\") pod \"315213c7-2b98-4a29-9239-76401beb51f7\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") "
Jul 28 17:31:30.003711 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.003695    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls\") pod \"315213c7-2b98-4a29-9239-76401beb51f7\" (UID: \"315213c7-2b98-4a29-9239-76401beb51f7\") "
Jul 28 17:31:30.003868 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.003850    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "315213c7-2b98-4a29-9239-76401beb51f7" (UID: "315213c7-2b98-4a29-9239-76401beb51f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:31:30.004022 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.003983    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/315213c7-2b98-4a29-9239-76401beb51f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:30.005375 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.005350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n" (OuterVolumeSpecName: "kube-api-access-vd52n") pod "315213c7-2b98-4a29-9239-76401beb51f7" (UID: "315213c7-2b98-4a29-9239-76401beb51f7"). InnerVolumeSpecName "kube-api-access-vd52n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:30.005506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.005489    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "315213c7-2b98-4a29-9239-76401beb51f7" (UID: "315213c7-2b98-4a29-9239-76401beb51f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:30.105097 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.105069    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd52n\" (UniqueName: \"kubernetes.io/projected/315213c7-2b98-4a29-9239-76401beb51f7-kube-api-access-vd52n\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:30.105097 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.105091    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/315213c7-2b98-4a29-9239-76401beb51f7-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:31:30.279920 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.279893    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:31:30.284362 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:30.284337    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1fbfc-predictor-6c94d86f78-pftbp"]
Jul 28 17:31:31.971954 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:31.971919    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="315213c7-2b98-4a29-9239-76401beb51f7" path="/var/lib/kubelet/pods/315213c7-2b98-4a29-9239-76401beb51f7/volumes"
Jul 28 17:31:31.972369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:31.972355    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c17a24-db85-4298-a8f4-762c1077759e" path="/var/lib/kubelet/pods/73c17a24-db85-4298-a8f4-762c1077759e/volumes"
Jul 28 17:31:32.949377 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:32.949347    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:31:32.950074 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:32.949896    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:31:33.954153 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:33.954117    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:31:33.954707 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:33.954678    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:42.950247 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:42.950205    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:31:43.955686 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:43.955644    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:31:52.950293 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:52.950245    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:31:53.955348 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:31:53.955305    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:02.950521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:02.950482    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 17:32:03.954895 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:03.954851    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 17:32:12.950481 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:12.950452    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:32:13.956012 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:13.955972    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:32:21.384642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.384613    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:21.507493 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.507452    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:32:21.507892 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.507843    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container" containerID="cri-o://30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83" gracePeriod=30
Jul 28 17:32:21.508173 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.508134    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kube-rbac-proxy" containerID="cri-o://6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c" gracePeriod=30
Jul 28 17:32:21.513135 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.513108    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:21.555827 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.555799    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:32:21.556072 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.556029    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container" containerID="cri-o://515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886" gracePeriod=30
Jul 28 17:32:21.556209 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.556069    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kube-rbac-proxy" containerID="cri-o://b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834" gracePeriod=30
Jul 28 17:32:21.583231 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.583205    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:21.599231 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599205    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:32:21.599532 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599520    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599534    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599545    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599550    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599561    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container"
Jul 28 17:32:21.599573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599567    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599581    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599586    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599636    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599644    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kserve-container"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599651    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="315213c7-2b98-4a29-9239-76401beb51f7" containerName="kube-rbac-proxy"
Jul 28 17:32:21.599719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.599656    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c17a24-db85-4298-a8f4-762c1077759e" containerName="kserve-container"
Jul 28 17:32:21.604299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.604274    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.606999 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.606962    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1bf99-predictor-serving-cert\""
Jul 28 17:32:21.607205 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.607192    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1bf99-kube-rbac-proxy-sar-config\""
Jul 28 17:32:21.611267 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.611244    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:32:21.677919 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.677869    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2lm\" (UniqueName: \"kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.678075 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.678036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.678144 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.678067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.678611 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.678590    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:21.697362 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.697339    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:32:21.700735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.700721    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.703386 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.703369    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1bf99-predictor-serving-cert\""
Jul 28 17:32:21.703479 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.703386    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1bf99-kube-rbac-proxy-sar-config\""
Jul 28 17:32:21.710891 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.710870    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:32:21.777921 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.777893    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:21.779030 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.778982    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.779140 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779036    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.779140 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779074    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.779140 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:21.779097    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-serving-cert: secret "success-200-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:21.779260 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:21.779153    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls podName:cba979ea-2b42-440f-8b13-493dff0c9bbd nodeName:}" failed. No retries permitted until 2026-07-28 17:32:22.279138998 +0000 UTC m=+1322.831448866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls") pod "success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" (UID: "cba979ea-2b42-440f-8b13-493dff0c9bbd") : secret "success-200-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:21.779260 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779104    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwzh\" (UniqueName: \"kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.779260 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.779260 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779231    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2lm\" (UniqueName: \"kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.779688 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.779672    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.787225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.787197    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2lm\" (UniqueName: \"kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:21.880444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.880418    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.880575 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.880499    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.880575 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.880529    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwzh\" (UniqueName: \"kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.880575 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:21.880556    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-serving-cert: secret "error-404-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:21.880757 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:21.880612    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls podName:503a4b91-47f6-43c2-a877-d021c1f1ca7b nodeName:}" failed. No retries permitted until 2026-07-28 17:32:22.380596768 +0000 UTC m=+1322.932906640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls") pod "error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" (UID: "503a4b91-47f6-43c2-a877-d021c1f1ca7b") : secret "error-404-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:21.881174 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.881153    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.888974 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.888950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwzh\" (UniqueName: \"kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:21.975293 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:21.975268    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:22.183222 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.183194    2552 generic.go:356] "Generic (PLEG): container finished" podID="77284c84-3f73-41eb-9ab2-352a44895984" containerID="b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834" exitCode=2
Jul 28 17:32:22.183366 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.183269    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerDied","Data":"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"}
Jul 28 17:32:22.184760 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.184737    2552 generic.go:356] "Generic (PLEG): container finished" podID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerID="6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c" exitCode=2
Jul 28 17:32:22.184871 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.184782    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerDied","Data":"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"}
Jul 28 17:32:22.282588 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.282527    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:22.285015 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.284976    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") pod \"success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") " pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:22.383073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.383045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:22.383229 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:22.383182    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-serving-cert: secret "error-404-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:22.383291 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:22.383250    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls podName:503a4b91-47f6-43c2-a877-d021c1f1ca7b nodeName:}" failed. No retries permitted until 2026-07-28 17:32:23.383232401 +0000 UTC m=+1323.935542270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls") pod "error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" (UID: "503a4b91-47f6-43c2-a877-d021c1f1ca7b") : secret "error-404-isvc-1bf99-predictor-serving-cert" not found
Jul 28 17:32:22.515934 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.515900    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:22.619575 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.619545    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:22.647304 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:22.647280    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:32:22.649797 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:32:22.649764    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba979ea_2b42_440f_8b13_493dff0c9bbd.slice/crio-766fd8be25117c9ff843d16090cfa3af30c6ae3f8fe2b4e19e72d06911e25991 WatchSource:0}: Error finding container 766fd8be25117c9ff843d16090cfa3af30c6ae3f8fe2b4e19e72d06911e25991: Status 404 returned error can't find the container with id 766fd8be25117c9ff843d16090cfa3af30c6ae3f8fe2b4e19e72d06911e25991
Jul 28 17:32:23.190747 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.190710    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerStarted","Data":"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"}
Jul 28 17:32:23.190935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.190771    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:23.190935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.190802    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerStarted","Data":"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"}
Jul 28 17:32:23.190935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.190908    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:23.191096 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.190951    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerStarted","Data":"766fd8be25117c9ff843d16090cfa3af30c6ae3f8fe2b4e19e72d06911e25991"}
Jul 28 17:32:23.191918 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.191898    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:23.209568 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.209516    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podStartSLOduration=2.209499457 podStartE2EDuration="2.209499457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:23.209104367 +0000 UTC m=+1323.761414278" watchObservedRunningTime="2026-07-28 17:32:23.209499457 +0000 UTC m=+1323.761809347"
Jul 28 17:32:23.391286 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.391250    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:23.393578 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.393555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") " pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:23.511189 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.511162    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:23.640117 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:23.640083    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:32:24.196533 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196433    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerStarted","Data":"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"}
Jul 28 17:32:24.196533 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196521    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:24.196751 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196716    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:24.196800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196766    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerStarted","Data":"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"}
Jul 28 17:32:24.196800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196714    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:24.196890 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.196801    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerStarted","Data":"c65ad6b0c60e982418a8cd0b602a55165dbbfd52c1aa5f4641c99de128ae910e"}
Jul 28 17:32:24.197450 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.197426    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:32:24.216290 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.216228    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podStartSLOduration=3.216216279 podStartE2EDuration="3.216216279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:24.214040263 +0000 UTC m=+1324.766350154" watchObservedRunningTime="2026-07-28 17:32:24.216216279 +0000 UTC m=+1324.768526164"
Jul 28 17:32:24.780215 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.780181    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:32:24.782855 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.782838    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:32:24.902551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902466    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") pod \"77284c84-3f73-41eb-9ab2-352a44895984\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") "
Jul 28 17:32:24.902551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902516    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config\") pod \"ffd61427-c7ab-4d72-aa02-a8018c09322b\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") "
Jul 28 17:32:24.902551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902533    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjn76\" (UniqueName: \"kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76\") pod \"77284c84-3f73-41eb-9ab2-352a44895984\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") "
Jul 28 17:32:24.902551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls\") pod \"ffd61427-c7ab-4d72-aa02-a8018c09322b\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") "
Jul 28 17:32:24.902858 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902674    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj8t7\" (UniqueName: \"kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7\") pod \"ffd61427-c7ab-4d72-aa02-a8018c09322b\" (UID: \"ffd61427-c7ab-4d72-aa02-a8018c09322b\") "
Jul 28 17:32:24.902858 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902733    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config\") pod \"77284c84-3f73-41eb-9ab2-352a44895984\" (UID: \"77284c84-3f73-41eb-9ab2-352a44895984\") "
Jul 28 17:32:24.902979 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.902882    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffd61427-c7ab-4d72-aa02-a8018c09322b" (UID: "ffd61427-c7ab-4d72-aa02-a8018c09322b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:24.903167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.903145    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:24.903236 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.903189    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77284c84-3f73-41eb-9ab2-352a44895984" (UID: "77284c84-3f73-41eb-9ab2-352a44895984"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:24.904665 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.904640    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77284c84-3f73-41eb-9ab2-352a44895984" (UID: "77284c84-3f73-41eb-9ab2-352a44895984"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:24.904778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.904726    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76" (OuterVolumeSpecName: "kube-api-access-jjn76") pod "77284c84-3f73-41eb-9ab2-352a44895984" (UID: "77284c84-3f73-41eb-9ab2-352a44895984"). InnerVolumeSpecName "kube-api-access-jjn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:24.904938 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.904923    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffd61427-c7ab-4d72-aa02-a8018c09322b" (UID: "ffd61427-c7ab-4d72-aa02-a8018c09322b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:24.904983 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:24.904956    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7" (OuterVolumeSpecName: "kube-api-access-sj8t7") pod "ffd61427-c7ab-4d72-aa02-a8018c09322b" (UID: "ffd61427-c7ab-4d72-aa02-a8018c09322b"). InnerVolumeSpecName "kube-api-access-sj8t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:25.004472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.004444    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77284c84-3f73-41eb-9ab2-352a44895984-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:25.004472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.004466    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77284c84-3f73-41eb-9ab2-352a44895984-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:25.004472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.004476    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjn76\" (UniqueName: \"kubernetes.io/projected/77284c84-3f73-41eb-9ab2-352a44895984-kube-api-access-jjn76\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:25.004658 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.004484    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffd61427-c7ab-4d72-aa02-a8018c09322b-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:25.004658 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.004493    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj8t7\" (UniqueName: \"kubernetes.io/projected/ffd61427-c7ab-4d72-aa02-a8018c09322b-kube-api-access-sj8t7\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:25.207149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.207115    2552 generic.go:356] "Generic (PLEG): container finished" podID="77284c84-3f73-41eb-9ab2-352a44895984" containerID="515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886" exitCode=0
Jul 28 17:32:25.207333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.207169    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"
Jul 28 17:32:25.207333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.207194    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerDied","Data":"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"}
Jul 28 17:32:25.207333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.207274    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55" event={"ID":"77284c84-3f73-41eb-9ab2-352a44895984","Type":"ContainerDied","Data":"d24cd3801274db637503cbe761de6ba62a18a7c2aece9675328fdc3cc2f406eb"}
Jul 28 17:32:25.207333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.207315    2552 scope.go:117] "RemoveContainer" containerID="b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"
Jul 28 17:32:25.208789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.208771    2552 generic.go:356] "Generic (PLEG): container finished" podID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerID="30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83" exitCode=0
Jul 28 17:32:25.208885 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.208832    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerDied","Data":"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"}
Jul 28 17:32:25.208928 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.208883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26" event={"ID":"ffd61427-c7ab-4d72-aa02-a8018c09322b","Type":"ContainerDied","Data":"5bd4f77d0cb3713387840d87c075d349bb0c3124a5a7f697281aad8d7db5fac3"}
Jul 28 17:32:25.209020 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.208984    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"
Jul 28 17:32:25.209172 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.209138    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:32:25.215492 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.215474    2552 scope.go:117] "RemoveContainer" containerID="515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"
Jul 28 17:32:25.223052 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.223036    2552 scope.go:117] "RemoveContainer" containerID="b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"
Jul 28 17:32:25.223299 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:25.223281    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834\": container with ID starting with b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834 not found: ID does not exist" containerID="b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"
Jul 28 17:32:25.223358 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.223303    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834"} err="failed to get container status \"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834\": rpc error: code = NotFound desc = could not find container \"b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834\": container with ID starting with b250aea7a4e061fdd10e4b9fe66cdca169f2a0b8d071594cac6daa6396e3d834 not found: ID does not exist"
Jul 28 17:32:25.223358 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.223318    2552 scope.go:117] "RemoveContainer" containerID="515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"
Jul 28 17:32:25.223540 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:25.223520    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886\": container with ID starting with 515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886 not found: ID does not exist" containerID="515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"
Jul 28 17:32:25.223573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.223547    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886"} err="failed to get container status \"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886\": rpc error: code = NotFound desc = could not find container \"515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886\": container with ID starting with 515ba50dbf324ace65030df2d832d38c6249074628d0eb9413b5ef278362f886 not found: ID does not exist"
Jul 28 17:32:25.223573 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.223562    2552 scope.go:117] "RemoveContainer" containerID="6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"
Jul 28 17:32:25.230317 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.230300    2552 scope.go:117] "RemoveContainer" containerID="30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"
Jul 28 17:32:25.232976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.232959    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:32:25.235662 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.235646    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3a5ae-predictor-7888f995f6-fzw55"]
Jul 28 17:32:25.238106 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.238092    2552 scope.go:117] "RemoveContainer" containerID="6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"
Jul 28 17:32:25.238358 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:25.238342    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c\": container with ID starting with 6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c not found: ID does not exist" containerID="6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"
Jul 28 17:32:25.238422 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.238366    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c"} err="failed to get container status \"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c\": rpc error: code = NotFound desc = could not find container \"6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c\": container with ID starting with 6350ee2cda4a1581d73e916f32d5f3ecfb663c7dd858b347f0aad592aacd390c not found: ID does not exist"
Jul 28 17:32:25.238422 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.238386    2552 scope.go:117] "RemoveContainer" containerID="30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"
Jul 28 17:32:25.238660 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:25.238643    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83\": container with ID starting with 30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83 not found: ID does not exist" containerID="30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"
Jul 28 17:32:25.238852 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.238666    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83"} err="failed to get container status \"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83\": rpc error: code = NotFound desc = could not find container \"30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83\": container with ID starting with 30b072c39a0b4daad6c5d589bdb63b41211a9c5bc7b100232cba1dd9a0eb8d83 not found: ID does not exist"
Jul 28 17:32:25.246560 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.246542    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:32:25.254079 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.254057    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3a5ae-predictor-69d6694669-t9t26"]
Jul 28 17:32:25.881697 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.881667    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:25.972112 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.972073    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77284c84-3f73-41eb-9ab2-352a44895984" path="/var/lib/kubelet/pods/77284c84-3f73-41eb-9ab2-352a44895984/volumes"
Jul 28 17:32:25.972584 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:25.972565    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" path="/var/lib/kubelet/pods/ffd61427-c7ab-4d72-aa02-a8018c09322b/volumes"
Jul 28 17:32:26.067114 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:26.067083    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:29.200797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:29.200768    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:32:29.201219 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:29.201195    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:30.213240 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:30.213216    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:32:30.213684 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:30.213662    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:32:35.964569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:35.964542    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.067209 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.067179    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:32:36.067458 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.067433    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container" containerID="cri-o://e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e" gracePeriod=30
Jul 28 17:32:36.067548 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.067525    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kube-rbac-proxy" containerID="cri-o://3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215" gracePeriod=30
Jul 28 17:32:36.096066 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.096021    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.128115 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.128085    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:32:36.128357 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.128326    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container" containerID="cri-o://fa17883b6df4928719303fd3b25e400da73e03bbf04c0fb0faa971dc80db33ca" gracePeriod=30
Jul 28 17:32:36.128503 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.128364    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kube-rbac-proxy" containerID="cri-o://ab54b42ce625aec9953d57ce54b95e3aa5e0c24d7d8b7eefe0e58d888ee1cc40" gracePeriod=30
Jul 28 17:32:36.131418 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131395    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:32:36.131818 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131800    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kube-rbac-proxy"
Jul 28 17:32:36.131818 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131819    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kube-rbac-proxy"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131839    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131847    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131868    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kube-rbac-proxy"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131877    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kube-rbac-proxy"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131894    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131902    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container"
Jul 28 17:32:36.131987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.131986    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kube-rbac-proxy"
Jul 28 17:32:36.132369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.132019    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="77284c84-3f73-41eb-9ab2-352a44895984" containerName="kserve-container"
Jul 28 17:32:36.132369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.132030    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kserve-container"
Jul 28 17:32:36.132369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.132038    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffd61427-c7ab-4d72-aa02-a8018c09322b" containerName="kube-rbac-proxy"
Jul 28 17:32:36.136960 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.136934    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.139698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.139531    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fca3b-kube-rbac-proxy-sar-config\""
Jul 28 17:32:36.139698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.139544    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fca3b-predictor-serving-cert\""
Jul 28 17:32:36.146392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.146231    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:32:36.161734 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.161699    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.193147 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.193106    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.193323 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.193167    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.193323 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.193245    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrpm\" (UniqueName: \"kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.243651 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.243627    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.244197 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.244176    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:32:36.247619 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.247597    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.252791 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.252767    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fca3b-predictor-serving-cert\""
Jul 28 17:32:36.253111 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.253094    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fca3b-kube-rbac-proxy-sar-config\""
Jul 28 17:32:36.262050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.262025    2552 generic.go:356] "Generic (PLEG): container finished" podID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerID="ab54b42ce625aec9953d57ce54b95e3aa5e0c24d7d8b7eefe0e58d888ee1cc40" exitCode=2
Jul 28 17:32:36.262149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.262100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerDied","Data":"ab54b42ce625aec9953d57ce54b95e3aa5e0c24d7d8b7eefe0e58d888ee1cc40"}
Jul 28 17:32:36.264102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.264077    2552 generic.go:356] "Generic (PLEG): container finished" podID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerID="3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215" exitCode=2
Jul 28 17:32:36.264191 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.264106    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerDied","Data":"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"}
Jul 28 17:32:36.272122 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.272101    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:32:36.294542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294516    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrpm\" (UniqueName: \"kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.294643 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.294643 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294631    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.294731 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294680    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99cm\" (UniqueName: \"kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.294767 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294741    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.294831 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:36.294818    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-serving-cert: secret "success-200-isvc-fca3b-predictor-serving-cert" not found
Jul 28 17:32:36.294878 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.294828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.294878 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:36.294865    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls podName:be9ccb9c-ebd3-44b2-b685-d5e260c772a9 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:36.794851423 +0000 UTC m=+1337.347161293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls") pod "success-200-isvc-fca3b-predictor-666b4997d9-54hbv" (UID: "be9ccb9c-ebd3-44b2-b685-d5e260c772a9") : secret "success-200-isvc-fca3b-predictor-serving-cert" not found
Jul 28 17:32:36.295430 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.295414    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.317873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.317851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrpm\" (UniqueName: \"kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.395864 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.395830    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.396060 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.395894    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.396060 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.395915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99cm\" (UniqueName: \"kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.396060 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:36.396031    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-serving-cert: secret "error-404-isvc-fca3b-predictor-serving-cert" not found
Jul 28 17:32:36.396248 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:36.396118    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls podName:a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:36.896095926 +0000 UTC m=+1337.448405798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls") pod "error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" (UID: "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9") : secret "error-404-isvc-fca3b-predictor-serving-cert" not found
Jul 28 17:32:36.396510 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.396490    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.404108 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.404085    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99cm\" (UniqueName: \"kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.571569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.571484    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.673597 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.673548    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:36.800186 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.800153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.802456 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.802432    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") pod \"success-200-isvc-fca3b-predictor-666b4997d9-54hbv\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") " pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:36.901436 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.901342    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:36.903625 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:36.903607    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") pod \"error-404-isvc-fca3b-predictor-678fd4d868-5gwnh\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") " pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:37.054421 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.054377    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:37.160384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.160301    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:37.222254 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.222223    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:32:37.224333 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:32:37.224305    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9ccb9c_ebd3_44b2_b685_d5e260c772a9.slice/crio-e23e158faae34c4ef8aa61ee6f5ba8dd0d5984526dc3923399d3ae53ecbdbde4 WatchSource:0}: Error finding container e23e158faae34c4ef8aa61ee6f5ba8dd0d5984526dc3923399d3ae53ecbdbde4: Status 404 returned error can't find the container with id e23e158faae34c4ef8aa61ee6f5ba8dd0d5984526dc3923399d3ae53ecbdbde4
Jul 28 17:32:37.271554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.271504    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerStarted","Data":"e23e158faae34c4ef8aa61ee6f5ba8dd0d5984526dc3923399d3ae53ecbdbde4"}
Jul 28 17:32:37.302090 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.302031    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:32:37.304708 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:32:37.304684    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ac4ed0_512e_4fe0_a7fa_f17be9ba2af9.slice/crio-b70a92011442b4225108cc514b93657b700146714a2eedef4f226efaeb5732ca WatchSource:0}: Error finding container b70a92011442b4225108cc514b93657b700146714a2eedef4f226efaeb5732ca: Status 404 returned error can't find the container with id b70a92011442b4225108cc514b93657b700146714a2eedef4f226efaeb5732ca
Jul 28 17:32:37.943936 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:37.943900    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 17:32:38.277384 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.277350    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerStarted","Data":"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"}
Jul 28 17:32:38.277826 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.277411    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerStarted","Data":"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"}
Jul 28 17:32:38.277826 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.277442    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerStarted","Data":"b70a92011442b4225108cc514b93657b700146714a2eedef4f226efaeb5732ca"}
Jul 28 17:32:38.277826 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.277504    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:38.279046 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.279024    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerStarted","Data":"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"}
Jul 28 17:32:38.279148 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.279077    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerStarted","Data":"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"}
Jul 28 17:32:38.279148 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.279107    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:38.279148 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.279126    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:38.280236 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.280213    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:32:38.297339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.297299    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podStartSLOduration=2.297286708 podStartE2EDuration="2.297286708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:38.295366114 +0000 UTC m=+1338.847676005" watchObservedRunningTime="2026-07-28 17:32:38.297286708 +0000 UTC m=+1338.849596598"
Jul 28 17:32:38.313293 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.313253    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podStartSLOduration=2.313240985 podStartE2EDuration="2.313240985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:38.312483454 +0000 UTC m=+1338.864793344" watchObservedRunningTime="2026-07-28 17:32:38.313240985 +0000 UTC m=+1338.865550876"
Jul 28 17:32:38.950735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:38.950686    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 17:32:39.201276 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.201196    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:39.285211 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.285179    2552 generic.go:356] "Generic (PLEG): container finished" podID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerID="fa17883b6df4928719303fd3b25e400da73e03bbf04c0fb0faa971dc80db33ca" exitCode=0
Jul 28 17:32:39.285642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.285257    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerDied","Data":"fa17883b6df4928719303fd3b25e400da73e03bbf04c0fb0faa971dc80db33ca"}
Jul 28 17:32:39.285715 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.285636    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:32:39.285769 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.285728    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:39.286906 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.286881    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:32:39.380338 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.380314    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:32:39.523886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.523852    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config\") pod \"46d8f25f-49af-4e5c-96cf-fef30bc44530\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") "
Jul 28 17:32:39.523886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.523888    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls\") pod \"46d8f25f-49af-4e5c-96cf-fef30bc44530\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") "
Jul 28 17:32:39.524104 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.523913    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nd27\" (UniqueName: \"kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27\") pod \"46d8f25f-49af-4e5c-96cf-fef30bc44530\" (UID: \"46d8f25f-49af-4e5c-96cf-fef30bc44530\") "
Jul 28 17:32:39.524230 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.524208    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46d8f25f-49af-4e5c-96cf-fef30bc44530" (UID: "46d8f25f-49af-4e5c-96cf-fef30bc44530"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:39.525967 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.525943    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46d8f25f-49af-4e5c-96cf-fef30bc44530" (UID: "46d8f25f-49af-4e5c-96cf-fef30bc44530"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:39.526103 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.526073    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27" (OuterVolumeSpecName: "kube-api-access-2nd27") pod "46d8f25f-49af-4e5c-96cf-fef30bc44530" (UID: "46d8f25f-49af-4e5c-96cf-fef30bc44530"). InnerVolumeSpecName "kube-api-access-2nd27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:39.624617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.624582    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:39.624617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.624610    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46d8f25f-49af-4e5c-96cf-fef30bc44530-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:39.624617 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.624622    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nd27\" (UniqueName: \"kubernetes.io/projected/46d8f25f-49af-4e5c-96cf-fef30bc44530-kube-api-access-2nd27\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:39.806150 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.806127    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:32:39.926206 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.926171    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls\") pod \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") "
Jul 28 17:32:39.926373 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.926239    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config\") pod \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") "
Jul 28 17:32:39.926373 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.926282    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmtx\" (UniqueName: \"kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx\") pod \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\" (UID: \"9f2a77b3-e9a3-4590-af3e-4f5477b72454\") "
Jul 28 17:32:39.926551 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.926530    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f2a77b3-e9a3-4590-af3e-4f5477b72454" (UID: "9f2a77b3-e9a3-4590-af3e-4f5477b72454"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:39.928303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.928263    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx" (OuterVolumeSpecName: "kube-api-access-jhmtx") pod "9f2a77b3-e9a3-4590-af3e-4f5477b72454" (UID: "9f2a77b3-e9a3-4590-af3e-4f5477b72454"). InnerVolumeSpecName "kube-api-access-jhmtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:39.928355 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:39.928331    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f2a77b3-e9a3-4590-af3e-4f5477b72454" (UID: "9f2a77b3-e9a3-4590-af3e-4f5477b72454"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:40.027687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.027658    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:40.027687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.027688    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmtx\" (UniqueName: \"kubernetes.io/projected/9f2a77b3-e9a3-4590-af3e-4f5477b72454-kube-api-access-jhmtx\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:40.027844 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.027703    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f2a77b3-e9a3-4590-af3e-4f5477b72454-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:32:40.213774 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.213741    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:32:40.291086 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.291049    2552 generic.go:356] "Generic (PLEG): container finished" podID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerID="e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e" exitCode=0
Jul 28 17:32:40.291520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.291107    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"
Jul 28 17:32:40.291520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.291135    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerDied","Data":"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"}
Jul 28 17:32:40.291520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.291193    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh" event={"ID":"9f2a77b3-e9a3-4590-af3e-4f5477b72454","Type":"ContainerDied","Data":"0989dd4f6d91a006c704de00825138cf853476f95a1c8fec06a8d635870a70b8"}
Jul 28 17:32:40.291520 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.291229    2552 scope.go:117] "RemoveContainer" containerID="3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"
Jul 28 17:32:40.292920 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.292901    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"
Jul 28 17:32:40.293105 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.292929    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk" event={"ID":"46d8f25f-49af-4e5c-96cf-fef30bc44530","Type":"ContainerDied","Data":"0f90701a0d2da3c6441e7b97c388a0df93f4e37e4c23fa375e9027fe05092ee5"}
Jul 28 17:32:40.293574 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.293399    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:32:40.299523 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.299506    2552 scope.go:117] "RemoveContainer" containerID="e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"
Jul 28 17:32:40.306849 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.306833    2552 scope.go:117] "RemoveContainer" containerID="3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"
Jul 28 17:32:40.308811 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:40.308779    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215\": container with ID starting with 3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215 not found: ID does not exist" containerID="3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"
Jul 28 17:32:40.308811 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.308805    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215"} err="failed to get container status \"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215\": rpc error: code = NotFound desc = could not find container \"3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215\": container with ID starting with 3200fe21db7291c1c2e60b442bf3e0cc65e11c35f4b64d3f751b1becb9161215 not found: ID does not exist"
Jul 28 17:32:40.308987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.308820    2552 scope.go:117] "RemoveContainer" containerID="e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"
Jul 28 17:32:40.309092 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:32:40.309066    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e\": container with ID starting with e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e not found: ID does not exist" containerID="e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"
Jul 28 17:32:40.309187 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.309093    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e"} err="failed to get container status \"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e\": rpc error: code = NotFound desc = could not find container \"e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e\": container with ID starting with e969512d759e6c115f0aa00adb5aad654e84c39ce2fd80a4cc7233921b841b8e not found: ID does not exist"
Jul 28 17:32:40.309187 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.309111    2552 scope.go:117] "RemoveContainer" containerID="ab54b42ce625aec9953d57ce54b95e3aa5e0c24d7d8b7eefe0e58d888ee1cc40"
Jul 28 17:32:40.314157 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.314131    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:32:40.320210 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.318081    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b0164-predictor-55bc6f9cdf-jknhh"]
Jul 28 17:32:40.320210 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.319269    2552 scope.go:117] "RemoveContainer" containerID="fa17883b6df4928719303fd3b25e400da73e03bbf04c0fb0faa971dc80db33ca"
Jul 28 17:32:40.330334 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.330309    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:32:40.333500 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:40.333482    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b0164-predictor-dd9648568-nwtkk"]
Jul 28 17:32:41.972240 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:41.972200    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" path="/var/lib/kubelet/pods/46d8f25f-49af-4e5c-96cf-fef30bc44530/volumes"
Jul 28 17:32:41.972647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:41.972635    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" path="/var/lib/kubelet/pods/9f2a77b3-e9a3-4590-af3e-4f5477b72454/volumes"
Jul 28 17:32:44.290474 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:44.290448    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:32:44.290975 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:44.290950    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:32:45.297303 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:45.297277    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:32:45.297776 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:45.297749    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:32:49.202062 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:49.202025    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:32:50.214428 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:50.214386    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:32:54.291545 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:54.291508    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:32:55.298401 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:55.298364    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:32:59.201257 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:32:59.201218    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:00.214308 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:00.214262    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:33:04.291656 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:04.291622    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:05.298331 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:05.298292    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:33:09.201179 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:09.201139    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 17:33:10.213861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:10.213820    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 17:33:14.290987 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:14.290941    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:33:15.297692 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:15.297656    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:33:19.202245 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:19.202211    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:33:20.214697 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:20.214671    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:33:24.291821 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:24.291791    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:33:25.298750 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:25.298704    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 17:33:31.651059 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:31.651021    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:31.800374 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:31.800343    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:35.298149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:35.298123    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:33:36.204088 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:36.204059    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:36.545164 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:36.545083    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:45.726610 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.726574    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:45.823713 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.823676    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:33:45.824106 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.824051    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container" containerID="cri-o://80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd" gracePeriod=30
Jul 28 17:33:45.824226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.824072    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kube-rbac-proxy" containerID="cri-o://d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0" gracePeriod=30
Jul 28 17:33:45.845533 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.845476    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:45.880642 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.880606    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:33:45.880925 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.880895    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container" containerID="cri-o://28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050" gracePeriod=30
Jul 28 17:33:45.881109 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.880930    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kube-rbac-proxy" containerID="cri-o://bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0" gracePeriod=30
Jul 28 17:33:45.897779 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.897755    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:33:45.898151 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898138    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898154    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898179    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898185    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898192    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898197    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container"
Jul 28 17:33:45.898207 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898205    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898210    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898262    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kserve-container"
Jul 28 17:33:45.898400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898271    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2a77b3-e9a3-4590-af3e-4f5477b72454" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898278    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kube-rbac-proxy"
Jul 28 17:33:45.898400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.898285    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="46d8f25f-49af-4e5c-96cf-fef30bc44530" containerName="kserve-container"
Jul 28 17:33:45.903251 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.903223    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:45.905976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.905951    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d7bb-predictor-serving-cert\""
Jul 28 17:33:45.905976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.905960    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d7bb-kube-rbac-proxy-sar-config\""
Jul 28 17:33:45.912439 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.912417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:33:45.916362 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.916341    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:45.960568 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.960532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:45.960732 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.960586    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:45.960732 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.960624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pbh\" (UniqueName: \"kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:45.983322 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.983241    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:45.994050 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.994013    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:33:45.997558 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:45.997536    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.000337 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.000317    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d7bb-predictor-serving-cert\""
Jul 28 17:33:46.000436 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.000344    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9d7bb-kube-rbac-proxy-sar-config\""
Jul 28 17:33:46.006817 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.006788    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:33:46.061461 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061431    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.061461 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061476    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061501    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061519    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pbh\" (UniqueName: \"kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9fp\" (UniqueName: \"kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.061571    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:46.061594    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-serving-cert: secret "success-200-isvc-9d7bb-predictor-serving-cert" not found
Jul 28 17:33:46.061739 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:46.061667    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls podName:8174e7bd-0b17-4167-a5f0-28e6398e6f74 nodeName:}" failed. No retries permitted until 2026-07-28 17:33:46.561649674 +0000 UTC m=+1407.113959543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls") pod "success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" (UID: "8174e7bd-0b17-4167-a5f0-28e6398e6f74") : secret "success-200-isvc-9d7bb-predictor-serving-cert" not found
Jul 28 17:33:46.062152 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.062133    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.072223 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.072186    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pbh\" (UniqueName: \"kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.147225 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.147187    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:46.162370 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.162341    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.162539 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.162375    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9fp\" (UniqueName: \"kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.162539 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.162395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.163064 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.163041    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.164935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.164911    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.170916 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.170896    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9fp\" (UniqueName: \"kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp\") pod \"error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") " pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.249525 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.249439    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:33:46.308782 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.308746    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.447735 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.447694    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:33:46.450567 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:33:46.450538    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda935516e_f8e9_47d4_94a2_c90e00577ddb.slice/crio-622ecfa0f8de506a942916a65991d9dd991ff574b6a8e196227666ebbd5d24d1 WatchSource:0}: Error finding container 622ecfa0f8de506a942916a65991d9dd991ff574b6a8e196227666ebbd5d24d1: Status 404 returned error can't find the container with id 622ecfa0f8de506a942916a65991d9dd991ff574b6a8e196227666ebbd5d24d1
Jul 28 17:33:46.566181 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.566158    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.568286 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.568265    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") pod \"success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") " pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.589612 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.589581    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerStarted","Data":"3291acf97d5debfcdb80ad58727a28af25f7d4a6310b3e0c937b4b2baf854403"}
Jul 28 17:33:46.589736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.589647    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerStarted","Data":"536c798dd86de5bec355b78ea00f16ae149a9524f360264c2f3a5965ea2a4433"}
Jul 28 17:33:46.589736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.589679    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerStarted","Data":"622ecfa0f8de506a942916a65991d9dd991ff574b6a8e196227666ebbd5d24d1"}
Jul 28 17:33:46.589736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.589713    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:46.591259 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.591240    2552 generic.go:356] "Generic (PLEG): container finished" podID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerID="d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0" exitCode=2
Jul 28 17:33:46.591368 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.591296    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerDied","Data":"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"}
Jul 28 17:33:46.592809 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.592792    2552 generic.go:356] "Generic (PLEG): container finished" podID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerID="bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0" exitCode=2
Jul 28 17:33:46.592887 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.592866    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerDied","Data":"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"}
Jul 28 17:33:46.610614 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.610574    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podStartSLOduration=1.610561058 podStartE2EDuration="1.610561058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:46.60959748 +0000 UTC m=+1407.161907372" watchObservedRunningTime="2026-07-28 17:33:46.610561058 +0000 UTC m=+1407.162870949"
Jul 28 17:33:46.815078 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.814979    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:46.945126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:46.945101    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:33:46.946727 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:33:46.946699    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8174e7bd_0b17_4167_a5f0_28e6398e6f74.slice/crio-97f524615c87a7f4f8a1d569ba845fa980da7aa9b8b46089ffbb8ca9aa3aaf84 WatchSource:0}: Error finding container 97f524615c87a7f4f8a1d569ba845fa980da7aa9b8b46089ffbb8ca9aa3aaf84: Status 404 returned error can't find the container with id 97f524615c87a7f4f8a1d569ba845fa980da7aa9b8b46089ffbb8ca9aa3aaf84
Jul 28 17:33:47.598923 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.598886    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerStarted","Data":"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"}
Jul 28 17:33:47.599121 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.598945    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerStarted","Data":"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"}
Jul 28 17:33:47.599224 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.599205    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:47.599287 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.599259    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerStarted","Data":"97f524615c87a7f4f8a1d569ba845fa980da7aa9b8b46089ffbb8ca9aa3aaf84"}
Jul 28 17:33:47.599333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.599290    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:47.599333 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.599312    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:47.600237 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.600203    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:33:47.600348 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.600290    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:33:47.617746 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:47.617704    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podStartSLOduration=2.617692592 podStartE2EDuration="2.617692592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:47.615537687 +0000 UTC m=+1408.167847575" watchObservedRunningTime="2026-07-28 17:33:47.617692592 +0000 UTC m=+1408.170002552"
Jul 28 17:33:48.604464 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:48.604420    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:33:48.604925 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:48.604502    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:33:49.046724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.046700    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:33:49.159156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.159132    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:33:49.191722 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.191692    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dwzh\" (UniqueName: \"kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh\") pod \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") "
Jul 28 17:33:49.191884 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.191747    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") pod \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") "
Jul 28 17:33:49.191884 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.191781    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config\") pod \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\" (UID: \"503a4b91-47f6-43c2-a877-d021c1f1ca7b\") "
Jul 28 17:33:49.192182 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.192156    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "503a4b91-47f6-43c2-a877-d021c1f1ca7b" (UID: "503a4b91-47f6-43c2-a877-d021c1f1ca7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:49.193763 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.193744    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "503a4b91-47f6-43c2-a877-d021c1f1ca7b" (UID: "503a4b91-47f6-43c2-a877-d021c1f1ca7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:49.193856 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.193759    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh" (OuterVolumeSpecName: "kube-api-access-9dwzh") pod "503a4b91-47f6-43c2-a877-d021c1f1ca7b" (UID: "503a4b91-47f6-43c2-a877-d021c1f1ca7b"). InnerVolumeSpecName "kube-api-access-9dwzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:49.292583 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292552    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") pod \"cba979ea-2b42-440f-8b13-493dff0c9bbd\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") "
Jul 28 17:33:49.292714 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292590    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config\") pod \"cba979ea-2b42-440f-8b13-493dff0c9bbd\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") "
Jul 28 17:33:49.292714 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292626    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn2lm\" (UniqueName: \"kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm\") pod \"cba979ea-2b42-440f-8b13-493dff0c9bbd\" (UID: \"cba979ea-2b42-440f-8b13-493dff0c9bbd\") "
Jul 28 17:33:49.292826 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292814    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwzh\" (UniqueName: \"kubernetes.io/projected/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-api-access-9dwzh\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.292862 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292829    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/503a4b91-47f6-43c2-a877-d021c1f1ca7b-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.292862 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292839    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/503a4b91-47f6-43c2-a877-d021c1f1ca7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.292981 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.292953    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cba979ea-2b42-440f-8b13-493dff0c9bbd" (UID: "cba979ea-2b42-440f-8b13-493dff0c9bbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:33:49.294569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.294553    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cba979ea-2b42-440f-8b13-493dff0c9bbd" (UID: "cba979ea-2b42-440f-8b13-493dff0c9bbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:49.294631 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.294552    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm" (OuterVolumeSpecName: "kube-api-access-tn2lm") pod "cba979ea-2b42-440f-8b13-493dff0c9bbd" (UID: "cba979ea-2b42-440f-8b13-493dff0c9bbd"). InnerVolumeSpecName "kube-api-access-tn2lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:49.393765 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.393720    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2lm\" (UniqueName: \"kubernetes.io/projected/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-api-access-tn2lm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.393765 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.393762    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cba979ea-2b42-440f-8b13-493dff0c9bbd-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.393765 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.393774    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cba979ea-2b42-440f-8b13-493dff0c9bbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:33:49.610786 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.610753    2552 generic.go:356] "Generic (PLEG): container finished" podID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerID="80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd" exitCode=0
Jul 28 17:33:49.611228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.610799    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"
Jul 28 17:33:49.611228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.610840    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerDied","Data":"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"}
Jul 28 17:33:49.611228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.610930    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4" event={"ID":"cba979ea-2b42-440f-8b13-493dff0c9bbd","Type":"ContainerDied","Data":"766fd8be25117c9ff843d16090cfa3af30c6ae3f8fe2b4e19e72d06911e25991"}
Jul 28 17:33:49.611228 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.610961    2552 scope.go:117] "RemoveContainer" containerID="d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"
Jul 28 17:33:49.612554 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.612526    2552 generic.go:356] "Generic (PLEG): container finished" podID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerID="28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050" exitCode=0
Jul 28 17:33:49.612640 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.612584    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"
Jul 28 17:33:49.612640 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.612610    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerDied","Data":"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"}
Jul 28 17:33:49.612728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.612674    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv" event={"ID":"503a4b91-47f6-43c2-a877-d021c1f1ca7b","Type":"ContainerDied","Data":"c65ad6b0c60e982418a8cd0b602a55165dbbfd52c1aa5f4641c99de128ae910e"}
Jul 28 17:33:49.619897 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.619884    2552 scope.go:117] "RemoveContainer" containerID="80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"
Jul 28 17:33:49.627609 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.627594    2552 scope.go:117] "RemoveContainer" containerID="d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"
Jul 28 17:33:49.627834 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:49.627816    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0\": container with ID starting with d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0 not found: ID does not exist" containerID="d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"
Jul 28 17:33:49.627886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.627841    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0"} err="failed to get container status \"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0\": rpc error: code = NotFound desc = could not find container \"d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0\": container with ID starting with d926d2fb48d5eb212b13c5f21364fe2a1a37ee3c26910a15ffb7914f1ca38dd0 not found: ID does not exist"
Jul 28 17:33:49.627886 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.627856    2552 scope.go:117] "RemoveContainer" containerID="80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"
Jul 28 17:33:49.628100 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:49.628079    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd\": container with ID starting with 80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd not found: ID does not exist" containerID="80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"
Jul 28 17:33:49.628159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.628105    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd"} err="failed to get container status \"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd\": rpc error: code = NotFound desc = could not find container \"80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd\": container with ID starting with 80e5e5cab8d0b011ae319b1f01d1298e3826041d10a167dd46b9f04b17df80bd not found: ID does not exist"
Jul 28 17:33:49.628159 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.628119    2552 scope.go:117] "RemoveContainer" containerID="bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"
Jul 28 17:33:49.634701 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.634680    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:33:49.638548 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.638527    2552 scope.go:117] "RemoveContainer" containerID="28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"
Jul 28 17:33:49.638848 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.638828    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1bf99-predictor-7c44b59b85-8kcd4"]
Jul 28 17:33:49.645700 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.645685    2552 scope.go:117] "RemoveContainer" containerID="bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"
Jul 28 17:33:49.645932 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:49.645912    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0\": container with ID starting with bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0 not found: ID does not exist" containerID="bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"
Jul 28 17:33:49.646051 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.645934    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0"} err="failed to get container status \"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0\": rpc error: code = NotFound desc = could not find container \"bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0\": container with ID starting with bd41f0f2ed6067a655f09bfe1c780b8cfe29381679c287fd8237458b286fcfd0 not found: ID does not exist"
Jul 28 17:33:49.646051 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.645949    2552 scope.go:117] "RemoveContainer" containerID="28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"
Jul 28 17:33:49.646196 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:33:49.646183    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050\": container with ID starting with 28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050 not found: ID does not exist" containerID="28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"
Jul 28 17:33:49.646253 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.646200    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050"} err="failed to get container status \"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050\": rpc error: code = NotFound desc = could not find container \"28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050\": container with ID starting with 28de4a7181dd4115ef955a1be027f023a20e1645c9c6766f41a214ba02f6e050 not found: ID does not exist"
Jul 28 17:33:49.648932 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.648910    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:33:49.652858 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.652829    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1bf99-predictor-5c574cdd84-pdwfv"]
Jul 28 17:33:49.974506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.974465    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" path="/var/lib/kubelet/pods/503a4b91-47f6-43c2-a877-d021c1f1ca7b/volumes"
Jul 28 17:33:49.974960 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:49.974941    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" path="/var/lib/kubelet/pods/cba979ea-2b42-440f-8b13-493dff0c9bbd/volumes"
Jul 28 17:33:53.608855 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:53.608825    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:33:53.609603 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:53.609572    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:33:53.610192 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:53.610172    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:33:53.610509 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:33:53.610489    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:34:03.610544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:03.610509    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:34:03.610544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:03.610523    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:34:10.649574 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:10.649543    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:13.609841 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:13.609759    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:34:13.610966 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:13.610933    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:34:23.610015 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:23.609962    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:34:23.611187 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:23.611165    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:34:33.611007 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:33.610967    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:34:33.611449 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:33.611137    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:34:46.697803 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:46.697761    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:46.753578 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:34:46.753546    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:35.854208 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:37:35.854132    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:15.733555 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:15.733526    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:50.917649 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:50.917610    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.042959 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.042927    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:41:51.043227 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.043202    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" containerID="cri-o://be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e" gracePeriod=30
Jul 28 17:41:51.043353 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.043312    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kube-rbac-proxy" containerID="cri-o://b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2" gracePeriod=30
Jul 28 17:41:51.066234 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.066205    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.118362 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118334    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:41:51.118673 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118661    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118674    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118690    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kube-rbac-proxy"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118696    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kube-rbac-proxy"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118702    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118708    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118718    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kube-rbac-proxy"
Jul 28 17:41:51.118728 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118723    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kube-rbac-proxy"
Jul 28 17:41:51.118954 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118778    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kube-rbac-proxy"
Jul 28 17:41:51.118954 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118787    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kserve-container"
Jul 28 17:41:51.118954 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118792    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="503a4b91-47f6-43c2-a877-d021c1f1ca7b" containerName="kserve-container"
Jul 28 17:41:51.118954 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.118800    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba979ea-2b42-440f-8b13-493dff0c9bbd" containerName="kube-rbac-proxy"
Jul 28 17:41:51.121815 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.121795    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.125107 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.125079    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5d39-predictor-serving-cert\""
Jul 28 17:41:51.125226 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.125088    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b5d39-kube-rbac-proxy-sar-config\""
Jul 28 17:41:51.132022 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.131965    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.133444 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.133399    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:41:51.159168 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.159139    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:41:51.159392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.159346    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container" containerID="cri-o://7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522" gracePeriod=30
Jul 28 17:41:51.159392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.159375    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kube-rbac-proxy" containerID="cri-o://295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d" gracePeriod=30
Jul 28 17:41:51.189091 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.189047    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.196489 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.196465    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:41:51.200024 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.199985    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.202789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.202773    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5d39-predictor-serving-cert\""
Jul 28 17:41:51.202864 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.202809    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b5d39-kube-rbac-proxy-sar-config\""
Jul 28 17:41:51.210694 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.210669    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:41:51.254975 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.254945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkgv\" (UniqueName: \"kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.255162 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.255011    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.255162 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.255119    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.290178 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.290158    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.356131 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356097    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.356268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356217    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.356268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356259    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.356385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkgv\" (UniqueName: \"kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.356385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qdz\" (UniqueName: \"kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.356468 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356396    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.357027 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.356982    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.358653 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.358636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.365529 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.365504    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkgv\" (UniqueName: \"kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv\") pod \"success-200-isvc-b5d39-predictor-6f5ff89745-nszhm\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") " pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.431874 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.431804    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:51.457179 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.457154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.457339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.457190    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qdz\" (UniqueName: \"kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.457339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.457254    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.457951 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.457931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.459506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.459486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.465786 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.465758    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qdz\" (UniqueName: \"kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz\") pod \"error-404-isvc-b5d39-predictor-84ff846864-69s5t\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") " pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.492962 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.492927    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:51.505131 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.505100    2552 generic.go:356] "Generic (PLEG): container finished" podID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerID="295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d" exitCode=2
Jul 28 17:41:51.505268 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.505168    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerDied","Data":"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"}
Jul 28 17:41:51.506910 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.506884    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerID="b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2" exitCode=2
Jul 28 17:41:51.507012 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.506924    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerDied","Data":"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"}
Jul 28 17:41:51.510101 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.510082    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:51.568547 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.568516    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:41:51.571647 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:41:51.571608    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5f73a8_86d6_4fb0_8a95_e4362eecd893.slice/crio-bc6f0d1da6f9583802873bb4b0594e07b00020924dd4ea12b85248ef7cbe2751 WatchSource:0}: Error finding container bc6f0d1da6f9583802873bb4b0594e07b00020924dd4ea12b85248ef7cbe2751: Status 404 returned error can't find the container with id bc6f0d1da6f9583802873bb4b0594e07b00020924dd4ea12b85248ef7cbe2751
Jul 28 17:41:51.647458 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:51.647437    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:41:51.649917 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:41:51.649895    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e48201_4e75_41a0_85b3_f5088eac37cb.slice/crio-141012c9fa1ce34ab87f16669611534f76217aa0b33d213076c303746db10cad WatchSource:0}: Error finding container 141012c9fa1ce34ab87f16669611534f76217aa0b33d213076c303746db10cad: Status 404 returned error can't find the container with id 141012c9fa1ce34ab87f16669611534f76217aa0b33d213076c303746db10cad
Jul 28 17:41:52.513073 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.513035    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerStarted","Data":"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"}
Jul 28 17:41:52.513564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.513115    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:52.513564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.513149    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:52.513564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.513175    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerStarted","Data":"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"}
Jul 28 17:41:52.513564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.513206    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerStarted","Data":"141012c9fa1ce34ab87f16669611534f76217aa0b33d213076c303746db10cad"}
Jul 28 17:41:52.514817 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.514789    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:52.514956 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.514931    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerStarted","Data":"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"}
Jul 28 17:41:52.515042 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.515002    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerStarted","Data":"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"}
Jul 28 17:41:52.515042 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.515029    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerStarted","Data":"bc6f0d1da6f9583802873bb4b0594e07b00020924dd4ea12b85248ef7cbe2751"}
Jul 28 17:41:52.515127 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.515063    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:52.534342 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.534297    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podStartSLOduration=1.534286522 podStartE2EDuration="1.534286522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:52.533826488 +0000 UTC m=+1893.086136378" watchObservedRunningTime="2026-07-28 17:41:52.534286522 +0000 UTC m=+1893.086596411"
Jul 28 17:41:52.553547 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:52.553458    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podStartSLOduration=1.553437906 podStartE2EDuration="1.553437906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:52.552042391 +0000 UTC m=+1893.104352302" watchObservedRunningTime="2026-07-28 17:41:52.553437906 +0000 UTC m=+1893.105747798"
Jul 28 17:41:53.520851 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:53.520803    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:53.521246 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:53.520953    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:53.522205 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:53.522177    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:41:54.286087 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:54.286049    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 17:41:54.291369 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:54.291347    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 17:41:54.525842 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:54.525796    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:41:54.921788 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:54.921764    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:41:55.003160 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.003106    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:41:55.089385 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.089355    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvrpm\" (UniqueName: \"kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm\") pod \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") "
Jul 28 17:41:55.089569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.089423    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config\") pod \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") "
Jul 28 17:41:55.089569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.089536    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") pod \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\" (UID: \"be9ccb9c-ebd3-44b2-b685-d5e260c772a9\") "
Jul 28 17:41:55.089804 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.089772    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be9ccb9c-ebd3-44b2-b685-d5e260c772a9" (UID: "be9ccb9c-ebd3-44b2-b685-d5e260c772a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:55.091470 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.091450    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be9ccb9c-ebd3-44b2-b685-d5e260c772a9" (UID: "be9ccb9c-ebd3-44b2-b685-d5e260c772a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:55.091559 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.091502    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm" (OuterVolumeSpecName: "kube-api-access-zvrpm") pod "be9ccb9c-ebd3-44b2-b685-d5e260c772a9" (UID: "be9ccb9c-ebd3-44b2-b685-d5e260c772a9"). InnerVolumeSpecName "kube-api-access-zvrpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:55.191033 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.190984    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d99cm\" (UniqueName: \"kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm\") pod \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") "
Jul 28 17:41:55.191033 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191038    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") pod \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") "
Jul 28 17:41:55.191218 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191073    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config\") pod \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\" (UID: \"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9\") "
Jul 28 17:41:55.191218 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191211    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.191286 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191229    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrpm\" (UniqueName: \"kubernetes.io/projected/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-api-access-zvrpm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.191286 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191239    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be9ccb9c-ebd3-44b2-b685-d5e260c772a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.191503 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.191472    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" (UID: "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:55.193095 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.193073    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" (UID: "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:55.193167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.193118    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm" (OuterVolumeSpecName: "kube-api-access-d99cm") pod "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" (UID: "a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9"). InnerVolumeSpecName "kube-api-access-d99cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:55.292026 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.291929    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99cm\" (UniqueName: \"kubernetes.io/projected/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-api-access-d99cm\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.292026 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.291957    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.292026 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.291968    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:41:55.531102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.531069    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerID="be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e" exitCode=0
Jul 28 17:41:55.531506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.531119    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"
Jul 28 17:41:55.531506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.531155    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerDied","Data":"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"}
Jul 28 17:41:55.531506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.531219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv" event={"ID":"be9ccb9c-ebd3-44b2-b685-d5e260c772a9","Type":"ContainerDied","Data":"e23e158faae34c4ef8aa61ee6f5ba8dd0d5984526dc3923399d3ae53ecbdbde4"}
Jul 28 17:41:55.531506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.531253    2552 scope.go:117] "RemoveContainer" containerID="b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"
Jul 28 17:41:55.532960 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.532940    2552 generic.go:356] "Generic (PLEG): container finished" podID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerID="7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522" exitCode=0
Jul 28 17:41:55.533081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.532975    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerDied","Data":"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"}
Jul 28 17:41:55.533081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.532983    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"
Jul 28 17:41:55.533081 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.533038    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh" event={"ID":"a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9","Type":"ContainerDied","Data":"b70a92011442b4225108cc514b93657b700146714a2eedef4f226efaeb5732ca"}
Jul 28 17:41:55.540736 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.540712    2552 scope.go:117] "RemoveContainer" containerID="be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"
Jul 28 17:41:55.549075 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.549059    2552 scope.go:117] "RemoveContainer" containerID="b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"
Jul 28 17:41:55.549334 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:41:55.549316    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2\": container with ID starting with b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2 not found: ID does not exist" containerID="b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"
Jul 28 17:41:55.549400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.549346    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2"} err="failed to get container status \"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2\": rpc error: code = NotFound desc = could not find container \"b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2\": container with ID starting with b9fda8992889c71bca47fbb34430d186b3a73613094f5dd122b2f647dec660b2 not found: ID does not exist"
Jul 28 17:41:55.549400 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.549361    2552 scope.go:117] "RemoveContainer" containerID="be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"
Jul 28 17:41:55.549611 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:41:55.549595    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e\": container with ID starting with be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e not found: ID does not exist" containerID="be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"
Jul 28 17:41:55.549659 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.549615    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e"} err="failed to get container status \"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e\": rpc error: code = NotFound desc = could not find container \"be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e\": container with ID starting with be59e78441e703ce821599927a63cd6b535c0dcbbcd3404a12f6f8fcc4d0e99e not found: ID does not exist"
Jul 28 17:41:55.549659 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.549628    2552 scope.go:117] "RemoveContainer" containerID="295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"
Jul 28 17:41:55.557149 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.557129    2552 scope.go:117] "RemoveContainer" containerID="7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"
Jul 28 17:41:55.557856 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.557835    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:41:55.562359 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.562340    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fca3b-predictor-666b4997d9-54hbv"]
Jul 28 17:41:55.564469 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.564448    2552 scope.go:117] "RemoveContainer" containerID="295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"
Jul 28 17:41:55.564717 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:41:55.564698    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d\": container with ID starting with 295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d not found: ID does not exist" containerID="295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"
Jul 28 17:41:55.564771 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.564724    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d"} err="failed to get container status \"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d\": rpc error: code = NotFound desc = could not find container \"295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d\": container with ID starting with 295b8cac722c13d8c1324e91348d1ae6b5e83d906a6dc86cdd69633c26a6f69d not found: ID does not exist"
Jul 28 17:41:55.564771 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.564740    2552 scope.go:117] "RemoveContainer" containerID="7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"
Jul 28 17:41:55.564962 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:41:55.564945    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522\": container with ID starting with 7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522 not found: ID does not exist" containerID="7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"
Jul 28 17:41:55.565025 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.564968    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522"} err="failed to get container status \"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522\": rpc error: code = NotFound desc = could not find container \"7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522\": container with ID starting with 7d8b2bd568ca3aa5f43682bac9bf7bc47ec5b4a453312fc8c41150eb81eff522 not found: ID does not exist"
Jul 28 17:41:55.573355 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.573322    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:41:55.574940 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.574921    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fca3b-predictor-678fd4d868-5gwnh"]
Jul 28 17:41:55.971797 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.971766    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" path="/var/lib/kubelet/pods/a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9/volumes"
Jul 28 17:41:55.972223 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:55.972194    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" path="/var/lib/kubelet/pods/be9ccb9c-ebd3-44b2-b685-d5e260c772a9/volumes"
Jul 28 17:41:58.525165 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:58.525137    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:41:58.525593 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:58.525568    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:41:59.530887 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:59.530857    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:41:59.531460 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:41:59.531431    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:42:08.526039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:08.525974    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:09.532217 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:09.532172    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:42:18.526443 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:18.526404    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:19.532133 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:19.532093    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:42:28.526258 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:28.526221    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 17:42:29.532356 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:29.532321    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 17:42:38.526139 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:38.526110    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:42:39.532030 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:39.531979    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:42:51.184516 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:51.184481    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:52.305381 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:42:52.305345    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:00.683511 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.683479    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:00.791100 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.791063    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:43:00.791425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.791401    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" containerID="cri-o://6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82" gracePeriod=30
Jul 28 17:43:00.791538 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.791406    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kube-rbac-proxy" containerID="cri-o://64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a" gracePeriod=30
Jul 28 17:43:00.814425 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.814403    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:00.840964 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.840935    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:43:00.841221 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.841196    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" containerID="cri-o://536c798dd86de5bec355b78ea00f16ae149a9524f360264c2f3a5965ea2a4433" gracePeriod=30
Jul 28 17:43:00.841325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.841237    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kube-rbac-proxy" containerID="cri-o://3291acf97d5debfcdb80ad58727a28af25f7d4a6310b3e0c937b4b2baf854403" gracePeriod=30
Jul 28 17:43:00.883569 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.883541    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:00.888798 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.888770    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 17:43:00.889339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889323    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container"
Jul 28 17:43:00.889392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889342    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container"
Jul 28 17:43:00.889392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889363    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889392 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889372    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889496 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889396    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889496 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889405    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889496 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889426    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container"
Jul 28 17:43:00.889496 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889434    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container"
Jul 28 17:43:00.889647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889520    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kserve-container"
Jul 28 17:43:00.889647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889532    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9ccb9c-ebd3-44b2-b685-d5e260c772a9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889542    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kube-rbac-proxy"
Jul 28 17:43:00.889647 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.889551    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ac4ed0-512e-4fe0-a7fa-f17be9ba2af9" containerName="kserve-container"
Jul 28 17:43:00.893040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.893023    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:00.895651 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.895630    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4209-predictor-serving-cert\""
Jul 28 17:43:00.895786 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.895766    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e4209-kube-rbac-proxy-sar-config\""
Jul 28 17:43:00.901217 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.901195    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 17:43:00.974068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.974043    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:00.991264 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.991236    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 17:43:00.994967 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.994951    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:00.997896 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.997876    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4209-kube-rbac-proxy-sar-config\""
Jul 28 17:43:00.998013 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:00.997874    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e4209-predictor-serving-cert\""
Jul 28 17:43:01.004378 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.004360    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 17:43:01.005641 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.005621    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chljj\" (UniqueName: \"kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.005765 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.005670    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.005765 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.005724    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.107039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107009    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.107039 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.107238 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107070    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.107238 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107162    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.107238 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107208    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chljj\" (UniqueName: \"kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.107358 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107238    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jgv6\" (UniqueName: \"kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.107672 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.107653    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.109585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.109562    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.112404 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.112371    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.119092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.119063    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chljj\" (UniqueName: \"kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj\") pod \"success-200-isvc-e4209-predictor-5d46777c7f-24bwf\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") " pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.205387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.205356    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.207685 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.207661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.207812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.207703    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jgv6\" (UniqueName: \"kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.207812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.207741    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.207812 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.207791    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e4209-predictor-serving-cert: secret "error-404-isvc-e4209-predictor-serving-cert" not found
Jul 28 17:43:01.207962 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.207847    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls podName:22f17563-00e2-47d2-91e8-662f3833f2f6 nodeName:}" failed. No retries permitted until 2026-07-28 17:43:01.707828521 +0000 UTC m=+1962.260138401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls") pod "error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" (UID: "22f17563-00e2-47d2-91e8-662f3833f2f6") : secret "error-404-isvc-e4209-predictor-serving-cert" not found
Jul 28 17:43:01.208377 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.208357    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.217752 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.217729    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jgv6\" (UniqueName: \"kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.265716 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.265650    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.346558 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.346531    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 17:43:01.349043 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:43:01.349011    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6906f40_d78c_436c_bd07_8d8b0ec81237.slice/crio-a7f582b9047c0f25387f7a4788dba1aa98e41f9bc05c41cfc30cb7d28b8b6331 WatchSource:0}: Error finding container a7f582b9047c0f25387f7a4788dba1aa98e41f9bc05c41cfc30cb7d28b8b6331: Status 404 returned error can't find the container with id a7f582b9047c0f25387f7a4788dba1aa98e41f9bc05c41cfc30cb7d28b8b6331
Jul 28 17:43:01.402658 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.402634    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:43:01.402862 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.402838    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container" containerID="cri-o://e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44" gracePeriod=30
Jul 28 17:43:01.402935 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.402864    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kube-rbac-proxy" containerID="cri-o://00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52" gracePeriod=30
Jul 28 17:43:01.417283 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.417262    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.500261 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.500144    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.506009 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.505964    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:43:01.509749 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.509707    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.512707 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.512670    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f50a-predictor-serving-cert\""
Jul 28 17:43:01.512861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.512681    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4f50a-kube-rbac-proxy-sar-config\""
Jul 28 17:43:01.520148 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.520073    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:43:01.560812 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.560788    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.611343 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.611310    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.611480 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.611348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.611542 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.611487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2k4\" (UniqueName: \"kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.625217 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.625197    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.628007 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.627973    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:43:01.631506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.631492    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.634567 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.634548    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f50a-predictor-serving-cert\""
Jul 28 17:43:01.634567 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.634559    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4f50a-kube-rbac-proxy-sar-config\""
Jul 28 17:43:01.643163 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.643144    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:43:01.681848 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.681814    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:43:01.682102 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.682069    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container" containerID="cri-o://c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444" gracePeriod=30
Jul 28 17:43:01.682247 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.682098    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kube-rbac-proxy" containerID="cri-o://c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc" gracePeriod=30
Jul 28 17:43:01.709630 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.709607    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.712044 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712025    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.712117 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712059    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.712117 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.712299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712120    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.712299 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712140    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ndk\" (UniqueName: \"kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.712299 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.712167    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-serving-cert: secret "success-200-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:01.712299 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.712229    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls podName:ea7b2898-7d16-4d1f-85ab-9c12e6e8508f nodeName:}" failed. No retries permitted until 2026-07-28 17:43:02.212210996 +0000 UTC m=+1962.764520866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls") pod "success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" (UID: "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f") : secret "success-200-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:01.712523 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.712523 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712400    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2k4\" (UniqueName: \"kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.712759 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.712740    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.714606 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.714585    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") pod \"error-404-isvc-e4209-predictor-595c5d94bb-tg9l4\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") " pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:01.723062 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.723044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2k4\" (UniqueName: \"kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:01.778129 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.778056    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:01.811130 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.811099    2552 generic.go:356] "Generic (PLEG): container finished" podID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerID="c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc" exitCode=2
Jul 28 17:43:01.811242 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.811177    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerDied","Data":"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"}
Jul 28 17:43:01.812852 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.812833    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ndk\" (UniqueName: \"kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.813010 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.812976    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.813094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerStarted","Data":"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"}
Jul 28 17:43:01.813094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813040    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.813094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813062    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerStarted","Data":"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"}
Jul 28 17:43:01.813094 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813094    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerStarted","Data":"a7f582b9047c0f25387f7a4788dba1aa98e41f9bc05c41cfc30cb7d28b8b6331"}
Jul 28 17:43:01.813355 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.813151    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-serving-cert: secret "error-404-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:01.813355 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813150    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:01.813355 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:01.813206    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls podName:fb6785bc-a8d9-4d36-b87e-df0bc221bdcc nodeName:}" failed. No retries permitted until 2026-07-28 17:43:02.313189267 +0000 UTC m=+1962.865499138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls") pod "error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" (UID: "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc") : secret "error-404-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:01.813763 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.813741    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.814932 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.814907    2552 generic.go:356] "Generic (PLEG): container finished" podID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerID="3291acf97d5debfcdb80ad58727a28af25f7d4a6310b3e0c937b4b2baf854403" exitCode=2
Jul 28 17:43:01.815026 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.814978    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerDied","Data":"3291acf97d5debfcdb80ad58727a28af25f7d4a6310b3e0c937b4b2baf854403"}
Jul 28 17:43:01.816800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.816779    2552 generic.go:356] "Generic (PLEG): container finished" podID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerID="64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a" exitCode=2
Jul 28 17:43:01.816899 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.816838    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerDied","Data":"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"}
Jul 28 17:43:01.818815 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.818796    2552 generic.go:356] "Generic (PLEG): container finished" podID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerID="00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52" exitCode=2
Jul 28 17:43:01.818895 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.818821    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerDied","Data":"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"}
Jul 28 17:43:01.824822 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.824804    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ndk\" (UniqueName: \"kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:01.837340 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.837298    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podStartSLOduration=1.8372854539999999 podStartE2EDuration="1.837285454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:01.835819645 +0000 UTC m=+1962.388129537" watchObservedRunningTime="2026-07-28 17:43:01.837285454 +0000 UTC m=+1962.389595388"
Jul 28 17:43:01.904848 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:01.904816    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:02.037523 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.037496    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 17:43:02.039542 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:43:02.039517    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f17563_00e2_47d2_91e8_662f3833f2f6.slice/crio-30a059100090df819f4ec1d45455b34e01880bf45c34d3009c19d413e915e5bd WatchSource:0}: Error finding container 30a059100090df819f4ec1d45455b34e01880bf45c34d3009c19d413e915e5bd: Status 404 returned error can't find the container with id 30a059100090df819f4ec1d45455b34e01880bf45c34d3009c19d413e915e5bd
Jul 28 17:43:02.216985 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.216956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:02.219111 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.219090    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") pod \"success-200-isvc-4f50a-predictor-569ccbb779-gwzcj\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") " pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:02.317574 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.317493    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:02.317711 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:02.317629    2552 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-serving-cert: secret "error-404-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:02.317711 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:02.317686    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls podName:fb6785bc-a8d9-4d36-b87e-df0bc221bdcc nodeName:}" failed. No retries permitted until 2026-07-28 17:43:03.317672027 +0000 UTC m=+1963.869981896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls") pod "error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" (UID: "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc") : secret "error-404-isvc-4f50a-predictor-serving-cert" not found
Jul 28 17:43:02.425281 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.425248    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:02.501545 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.501517    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:02.558030 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.558005    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:43:02.559888 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:43:02.559863    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7b2898_7d16_4d1f_85ab_9c12e6e8508f.slice/crio-33fb1070e665d3be83397d250dcda2321bb1abb124db44d9f6033198174d1a5b WatchSource:0}: Error finding container 33fb1070e665d3be83397d250dcda2321bb1abb124db44d9f6033198174d1a5b: Status 404 returned error can't find the container with id 33fb1070e665d3be83397d250dcda2321bb1abb124db44d9f6033198174d1a5b
Jul 28 17:43:02.824873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.824787    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerStarted","Data":"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"}
Jul 28 17:43:02.824873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.824842    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerStarted","Data":"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"}
Jul 28 17:43:02.824873 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.824873    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:02.825387 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.824905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerStarted","Data":"33fb1070e665d3be83397d250dcda2321bb1abb124db44d9f6033198174d1a5b"}
Jul 28 17:43:02.829219 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.829103    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerStarted","Data":"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"}
Jul 28 17:43:02.829219 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.829168    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerStarted","Data":"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"}
Jul 28 17:43:02.829219 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.829202    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerStarted","Data":"30a059100090df819f4ec1d45455b34e01880bf45c34d3009c19d413e915e5bd"}
Jul 28 17:43:02.829691 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.829668    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:02.830484 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.830453    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:02.847934 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.847879    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podStartSLOduration=1.847860168 podStartE2EDuration="1.847860168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:02.844821415 +0000 UTC m=+1963.397131306" watchObservedRunningTime="2026-07-28 17:43:02.847860168 +0000 UTC m=+1963.400170057"
Jul 28 17:43:02.867060 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:02.867018    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podStartSLOduration=2.866985612 podStartE2EDuration="2.866985612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:02.865568552 +0000 UTC m=+1963.417878443" watchObservedRunningTime="2026-07-28 17:43:02.866985612 +0000 UTC m=+1963.419295547"
Jul 28 17:43:03.328272 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.328215    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:03.330570 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.330542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"error-404-isvc-4f50a-predictor-54cf775f94-5gqtw\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") " pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:03.440912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.440873    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:03.521030 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.520969    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 17:43:03.584497 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.584472    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:43:03.587255 ip-10-0-136-80 kubenswrapper[2552]: W0728 17:43:03.587231    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6785bc_a8d9_4d36_b87e_df0bc221bdcc.slice/crio-901c15998dcd316efcd85c00f9caf6772d14b3c2b0df548130e0bac8ac6e52b5 WatchSource:0}: Error finding container 901c15998dcd316efcd85c00f9caf6772d14b3c2b0df548130e0bac8ac6e52b5: Status 404 returned error can't find the container with id 901c15998dcd316efcd85c00f9caf6772d14b3c2b0df548130e0bac8ac6e52b5
Jul 28 17:43:03.605212 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.605182    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 17:43:03.605324 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.605182    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 17:43:03.610080 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.610051    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 17:43:03.611014 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.610985    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 17:43:03.835335 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835250    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerStarted","Data":"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"}
Jul 28 17:43:03.835335 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835325    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerStarted","Data":"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"}
Jul 28 17:43:03.835778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835499    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:03.835778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835548    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerStarted","Data":"901c15998dcd316efcd85c00f9caf6772d14b3c2b0df548130e0bac8ac6e52b5"}
Jul 28 17:43:03.835778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835581    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:03.835778 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835614    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:03.835973 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835800    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:03.835973 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835842    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:03.836101 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.835969    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:03.836620 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.836596    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:03.836738 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.836679    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:03.836914 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.836900    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:03.857435 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:03.857372    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podStartSLOduration=2.85735114 podStartE2EDuration="2.85735114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:03.854695518 +0000 UTC m=+1964.407005408" watchObservedRunningTime="2026-07-28 17:43:03.85735114 +0000 UTC m=+1964.409661032"
Jul 28 17:43:04.526265 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.526235    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 17:43:04.644540 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.644518    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:43:04.739961 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.739929    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") pod \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") "
Jul 28 17:43:04.740139 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.739987    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57pbh\" (UniqueName: \"kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh\") pod \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") "
Jul 28 17:43:04.740139 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.740090    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config\") pod \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\" (UID: \"8174e7bd-0b17-4167-a5f0-28e6398e6f74\") "
Jul 28 17:43:04.740442 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.740418    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8174e7bd-0b17-4167-a5f0-28e6398e6f74" (UID: "8174e7bd-0b17-4167-a5f0-28e6398e6f74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:04.741933 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.741913    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh" (OuterVolumeSpecName: "kube-api-access-57pbh") pod "8174e7bd-0b17-4167-a5f0-28e6398e6f74" (UID: "8174e7bd-0b17-4167-a5f0-28e6398e6f74"). InnerVolumeSpecName "kube-api-access-57pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:04.742024 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.741951    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8174e7bd-0b17-4167-a5f0-28e6398e6f74" (UID: "8174e7bd-0b17-4167-a5f0-28e6398e6f74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:04.840698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.840668    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pbh\" (UniqueName: \"kubernetes.io/projected/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-api-access-57pbh\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:04.840698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.840691    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8174e7bd-0b17-4167-a5f0-28e6398e6f74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:04.840698 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.840702    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8174e7bd-0b17-4167-a5f0-28e6398e6f74-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:04.841548 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.841527    2552 generic.go:356] "Generic (PLEG): container finished" podID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerID="536c798dd86de5bec355b78ea00f16ae149a9524f360264c2f3a5965ea2a4433" exitCode=0
Jul 28 17:43:04.841629 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.841605    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerDied","Data":"536c798dd86de5bec355b78ea00f16ae149a9524f360264c2f3a5965ea2a4433"}
Jul 28 17:43:04.842973 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.842952    2552 generic.go:356] "Generic (PLEG): container finished" podID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerID="6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82" exitCode=0
Jul 28 17:43:04.843084 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843012    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"
Jul 28 17:43:04.843084 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843019    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerDied","Data":"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"}
Jul 28 17:43:04.843084 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843069    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt" event={"ID":"8174e7bd-0b17-4167-a5f0-28e6398e6f74","Type":"ContainerDied","Data":"97f524615c87a7f4f8a1d569ba845fa980da7aa9b8b46089ffbb8ca9aa3aaf84"}
Jul 28 17:43:04.843245 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843098    2552 scope.go:117] "RemoveContainer" containerID="64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"
Jul 28 17:43:04.843521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843495    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:04.843841 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843754    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:04.843930 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.843909    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:04.857500 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.857477    2552 scope.go:117] "RemoveContainer" containerID="6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"
Jul 28 17:43:04.865201 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.865181    2552 scope.go:117] "RemoveContainer" containerID="64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"
Jul 28 17:43:04.865459 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:04.865427    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a\": container with ID starting with 64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a not found: ID does not exist" containerID="64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"
Jul 28 17:43:04.865513 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.865467    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a"} err="failed to get container status \"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a\": rpc error: code = NotFound desc = could not find container \"64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a\": container with ID starting with 64bbefdf34f5ac7bf4ed39e3a6759e8eea76c5a9d078bcf99b146655a977478a not found: ID does not exist"
Jul 28 17:43:04.865513 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.865483    2552 scope.go:117] "RemoveContainer" containerID="6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"
Jul 28 17:43:04.865758 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:04.865742    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82\": container with ID starting with 6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82 not found: ID does not exist" containerID="6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"
Jul 28 17:43:04.865802 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.865768    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82"} err="failed to get container status \"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82\": rpc error: code = NotFound desc = could not find container \"6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82\": container with ID starting with 6613b80b839bbb0b4892e2352d5b66313539c2d2f626ad7e177d8eb1b0a4cc82 not found: ID does not exist"
Jul 28 17:43:04.873460 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.873423    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:43:04.877819 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:04.877799    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d7bb-predictor-7c9b6cdf9f-445tt"]
Jul 28 17:43:05.124650 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.124630    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:43:05.244719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.244694    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config\") pod \"a935516e-f8e9-47d4-94a2-c90e00577ddb\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") "
Jul 28 17:43:05.244854 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.244742    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz9fp\" (UniqueName: \"kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp\") pod \"a935516e-f8e9-47d4-94a2-c90e00577ddb\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") "
Jul 28 17:43:05.244854 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.244823    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls\") pod \"a935516e-f8e9-47d4-94a2-c90e00577ddb\" (UID: \"a935516e-f8e9-47d4-94a2-c90e00577ddb\") "
Jul 28 17:43:05.245098 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.245074    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a935516e-f8e9-47d4-94a2-c90e00577ddb" (UID: "a935516e-f8e9-47d4-94a2-c90e00577ddb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:05.247894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.247848    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp" (OuterVolumeSpecName: "kube-api-access-wz9fp") pod "a935516e-f8e9-47d4-94a2-c90e00577ddb" (UID: "a935516e-f8e9-47d4-94a2-c90e00577ddb"). InnerVolumeSpecName "kube-api-access-wz9fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:05.247894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.247875    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a935516e-f8e9-47d4-94a2-c90e00577ddb" (UID: "a935516e-f8e9-47d4-94a2-c90e00577ddb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:05.262871 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.262816    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:43:05.345671 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345649    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls\") pod \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") "
Jul 28 17:43:05.345789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345687    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrkgv\" (UniqueName: \"kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv\") pod \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") "
Jul 28 17:43:05.345789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345711    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config\") pod \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\" (UID: \"5c5f73a8-86d6-4fb0-8a95-e4362eecd893\") "
Jul 28 17:43:05.345936 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345916    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a935516e-f8e9-47d4-94a2-c90e00577ddb-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.345976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345942    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.345976 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.345957    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9fp\" (UniqueName: \"kubernetes.io/projected/a935516e-f8e9-47d4-94a2-c90e00577ddb-kube-api-access-wz9fp\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.346126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.346070    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c5f73a8-86d6-4fb0-8a95-e4362eecd893" (UID: "5c5f73a8-86d6-4fb0-8a95-e4362eecd893"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:05.347853 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.347826    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c5f73a8-86d6-4fb0-8a95-e4362eecd893" (UID: "5c5f73a8-86d6-4fb0-8a95-e4362eecd893"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:05.348005 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.347972    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv" (OuterVolumeSpecName: "kube-api-access-hrkgv") pod "5c5f73a8-86d6-4fb0-8a95-e4362eecd893" (UID: "5c5f73a8-86d6-4fb0-8a95-e4362eecd893"). InnerVolumeSpecName "kube-api-access-hrkgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:05.446894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.446852    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.446894 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.446890    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkgv\" (UniqueName: \"kubernetes.io/projected/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-api-access-hrkgv\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.447126 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.446906    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5f73a8-86d6-4fb0-8a95-e4362eecd893-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.615630 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.615608    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:43:05.749367 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.749332    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config\") pod \"b2e48201-4e75-41a0-85b3-f5088eac37cb\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") "
Jul 28 17:43:05.749585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.749394    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78qdz\" (UniqueName: \"kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz\") pod \"b2e48201-4e75-41a0-85b3-f5088eac37cb\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") "
Jul 28 17:43:05.749585 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.749443    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls\") pod \"b2e48201-4e75-41a0-85b3-f5088eac37cb\" (UID: \"b2e48201-4e75-41a0-85b3-f5088eac37cb\") "
Jul 28 17:43:05.749883 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.749857    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2e48201-4e75-41a0-85b3-f5088eac37cb" (UID: "b2e48201-4e75-41a0-85b3-f5088eac37cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:05.751616 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.751591    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2e48201-4e75-41a0-85b3-f5088eac37cb" (UID: "b2e48201-4e75-41a0-85b3-f5088eac37cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:05.751884 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.751865    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz" (OuterVolumeSpecName: "kube-api-access-78qdz") pod "b2e48201-4e75-41a0-85b3-f5088eac37cb" (UID: "b2e48201-4e75-41a0-85b3-f5088eac37cb"). InnerVolumeSpecName "kube-api-access-78qdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:05.850141 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850107    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.850141 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850135    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qdz\" (UniqueName: \"kubernetes.io/projected/b2e48201-4e75-41a0-85b3-f5088eac37cb-kube-api-access-78qdz\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.850608 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850150    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e48201-4e75-41a0-85b3-f5088eac37cb-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:43:05.850608 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850251    2552 generic.go:356] "Generic (PLEG): container finished" podID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerID="e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44" exitCode=0
Jul 28 17:43:05.850608 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850407    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"
Jul 28 17:43:05.850957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850801    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerDied","Data":"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"}
Jul 28 17:43:05.850957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850909    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm" event={"ID":"5c5f73a8-86d6-4fb0-8a95-e4362eecd893","Type":"ContainerDied","Data":"bc6f0d1da6f9583802873bb4b0594e07b00020924dd4ea12b85248ef7cbe2751"}
Jul 28 17:43:05.850957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.850929    2552 scope.go:117] "RemoveContainer" containerID="00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"
Jul 28 17:43:05.853366 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.853339    2552 generic.go:356] "Generic (PLEG): container finished" podID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerID="c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444" exitCode=0
Jul 28 17:43:05.853479 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.853406    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"
Jul 28 17:43:05.853544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.853479    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerDied","Data":"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"}
Jul 28 17:43:05.853544 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.853526    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t" event={"ID":"b2e48201-4e75-41a0-85b3-f5088eac37cb","Type":"ContainerDied","Data":"141012c9fa1ce34ab87f16669611534f76217aa0b33d213076c303746db10cad"}
Jul 28 17:43:05.856852 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.856827    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2" event={"ID":"a935516e-f8e9-47d4-94a2-c90e00577ddb","Type":"ContainerDied","Data":"622ecfa0f8de506a942916a65991d9dd991ff574b6a8e196227666ebbd5d24d1"}
Jul 28 17:43:05.856852 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.856848    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"
Jul 28 17:43:05.864739 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.864721    2552 scope.go:117] "RemoveContainer" containerID="e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"
Jul 28 17:43:05.872931 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.872916    2552 scope.go:117] "RemoveContainer" containerID="00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"
Jul 28 17:43:05.873177 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:05.873157    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52\": container with ID starting with 00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52 not found: ID does not exist" containerID="00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"
Jul 28 17:43:05.873233 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.873184    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52"} err="failed to get container status \"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52\": rpc error: code = NotFound desc = could not find container \"00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52\": container with ID starting with 00280a1339f7707c237e56b51f4433f9321564758e1e3aaf513ccadeecb72c52 not found: ID does not exist"
Jul 28 17:43:05.873233 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.873200    2552 scope.go:117] "RemoveContainer" containerID="e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"
Jul 28 17:43:05.873460 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:05.873428    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44\": container with ID starting with e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44 not found: ID does not exist" containerID="e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"
Jul 28 17:43:05.873499 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.873466    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44"} err="failed to get container status \"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44\": rpc error: code = NotFound desc = could not find container \"e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44\": container with ID starting with e4a1809b1608d354e604e05c28a5403199a1201370112cbb96cd6d22205f0e44 not found: ID does not exist"
Jul 28 17:43:05.873499 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.873481    2552 scope.go:117] "RemoveContainer" containerID="c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"
Jul 28 17:43:05.879851 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.879828    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:43:05.880861 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.880845    2552 scope.go:117] "RemoveContainer" containerID="c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"
Jul 28 17:43:05.884340 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.884317    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b5d39-predictor-6f5ff89745-nszhm"]
Jul 28 17:43:05.888070 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.888055    2552 scope.go:117] "RemoveContainer" containerID="c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"
Jul 28 17:43:05.888308 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:05.888291    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc\": container with ID starting with c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc not found: ID does not exist" containerID="c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"
Jul 28 17:43:05.888368 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.888312    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc"} err="failed to get container status \"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc\": rpc error: code = NotFound desc = could not find container \"c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc\": container with ID starting with c07f5c0013d4a938507d1ab3082bbb45e1147df5b94784d65f0421dca98f83dc not found: ID does not exist"
Jul 28 17:43:05.888368 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.888326    2552 scope.go:117] "RemoveContainer" containerID="c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"
Jul 28 17:43:05.888558 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:43:05.888538    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444\": container with ID starting with c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444 not found: ID does not exist" containerID="c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"
Jul 28 17:43:05.888600 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.888564    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444"} err="failed to get container status \"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444\": rpc error: code = NotFound desc = could not find container \"c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444\": container with ID starting with c90a7f44061e95baa9311e4f3a75f7c946fa67b29193fa88070dccc876010444 not found: ID does not exist"
Jul 28 17:43:05.888600 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.888579    2552 scope.go:117] "RemoveContainer" containerID="3291acf97d5debfcdb80ad58727a28af25f7d4a6310b3e0c937b4b2baf854403"
Jul 28 17:43:05.895349 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.895334    2552 scope.go:117] "RemoveContainer" containerID="536c798dd86de5bec355b78ea00f16ae149a9524f360264c2f3a5965ea2a4433"
Jul 28 17:43:05.899635 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.899616    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:43:05.902800 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.902781    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9d7bb-predictor-85c9b46cf4-wzht2"]
Jul 28 17:43:05.914789 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.914770    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:43:05.918830 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.918810    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b5d39-predictor-84ff846864-69s5t"]
Jul 28 17:43:05.971687 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.971659    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" path="/var/lib/kubelet/pods/5c5f73a8-86d6-4fb0-8a95-e4362eecd893/volumes"
Jul 28 17:43:05.972092 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.972079    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" path="/var/lib/kubelet/pods/8174e7bd-0b17-4167-a5f0-28e6398e6f74/volumes"
Jul 28 17:43:05.972486 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.972474    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" path="/var/lib/kubelet/pods/a935516e-f8e9-47d4-94a2-c90e00577ddb/volumes"
Jul 28 17:43:05.973156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:05.973143    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" path="/var/lib/kubelet/pods/b2e48201-4e75-41a0-85b3-f5088eac37cb/volumes"
Jul 28 17:43:08.840749 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:08.840719    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:08.841232 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:08.841209    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:09.849920 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.849380    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:09.849920 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.849766    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:43:09.849920 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.849904    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:09.850548 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.850150    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:09.850548 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.850154    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:09.850777 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:09.850755    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:18.841910 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:18.841830    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:19.850105 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:19.850056    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:19.850558 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:19.850155    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:19.851480 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:19.851449    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:28.841724 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:28.841679    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:29.850982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:29.850938    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:29.850982 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:29.850978    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:29.851521 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:29.851010    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:38.841339 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:38.841297    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:39.850248 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:39.850207    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 17:43:39.850641 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:39.850218    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:39.850781 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:39.850757    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 17:43:48.841374 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:48.841332    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 17:43:49.850200 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:49.850157    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 28 17:43:49.850622 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:49.850603    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 17:43:49.851285 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:49.851265    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:43:58.842040 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:58.842010    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 17:43:59.850564 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:43:59.850539    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:44:01.532024 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:01.531980    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:01.977095 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:01.977063    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:10.965830 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:10.965799    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:11.285280 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:11.285198    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:46.203506 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:46.203429    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:44:46.275086 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:44:46.275061    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:17.554113 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:46:17.554032    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:28.719114 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:47:28.719077    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:50:47.104472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:50:47.104396    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:16.205049 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.204937    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:16.336337 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.336307    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:52:16.336566 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.336541    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container" containerID="cri-o://40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f" gracePeriod=30
Jul 28 17:52:16.336636 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.336577    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kube-rbac-proxy" containerID="cri-o://2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33" gracePeriod=30
Jul 28 17:52:16.363472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.363446    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:16.385511 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.385481    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:52:16.385785 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.385757    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container" containerID="cri-o://9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3" gracePeriod=30
Jul 28 17:52:16.385929 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.385806    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kube-rbac-proxy" containerID="cri-o://46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c" gracePeriod=30
Jul 28 17:52:16.427850 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:16.427820    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:17.015914 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:17.015883    2552 generic.go:356] "Generic (PLEG): container finished" podID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerID="2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33" exitCode=2
Jul 28 17:52:17.016110 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:17.015955    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerDied","Data":"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"}
Jul 28 17:52:17.017701 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:17.017679    2552 generic.go:356] "Generic (PLEG): container finished" podID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerID="46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c" exitCode=2
Jul 28 17:52:17.017804 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:17.017710    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerDied","Data":"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"}
Jul 28 17:52:19.500819 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.500794    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:52:19.503725 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.503681    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:52:19.629872 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.629837    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config\") pod \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") "
Jul 28 17:52:19.630084 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.629917    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config\") pod \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") "
Jul 28 17:52:19.630167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630081    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5ndk\" (UniqueName: \"kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk\") pod \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") "
Jul 28 17:52:19.630167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630126    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") pod \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\" (UID: \"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc\") "
Jul 28 17:52:19.630167 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630159    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") pod \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") "
Jul 28 17:52:19.630325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630172    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" (UID: "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:19.630325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630188    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw2k4\" (UniqueName: \"kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4\") pod \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\" (UID: \"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f\") "
Jul 28 17:52:19.630325 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630222    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" (UID: "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:19.630472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630445    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:19.630472 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.630463    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:19.633825 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.632522    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4" (OuterVolumeSpecName: "kube-api-access-pw2k4") pod "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" (UID: "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f"). InnerVolumeSpecName "kube-api-access-pw2k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:19.637347 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.637318    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk" (OuterVolumeSpecName: "kube-api-access-c5ndk") pod "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" (UID: "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc"). InnerVolumeSpecName "kube-api-access-c5ndk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:19.637467 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.637383    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" (UID: "ea7b2898-7d16-4d1f-85ab-9c12e6e8508f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:19.637525 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.637434    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" (UID: "fb6785bc-a8d9-4d36-b87e-df0bc221bdcc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:19.731336 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.731303    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ndk\" (UniqueName: \"kubernetes.io/projected/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-kube-api-access-c5ndk\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:19.731336 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.731331    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:19.731508 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.731346    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:19.731508 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:19.731358    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw2k4\" (UniqueName: \"kubernetes.io/projected/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f-kube-api-access-pw2k4\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 17:52:20.030447 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.030415    2552 generic.go:356] "Generic (PLEG): container finished" podID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerID="40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f" exitCode=0
Jul 28 17:52:20.030621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.030476    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"
Jul 28 17:52:20.030621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.030502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerDied","Data":"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"}
Jul 28 17:52:20.030621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.030561    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj" event={"ID":"ea7b2898-7d16-4d1f-85ab-9c12e6e8508f","Type":"ContainerDied","Data":"33fb1070e665d3be83397d250dcda2321bb1abb124db44d9f6033198174d1a5b"}
Jul 28 17:52:20.030621 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.030588    2552 scope.go:117] "RemoveContainer" containerID="2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"
Jul 28 17:52:20.032344 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.032316    2552 generic.go:356] "Generic (PLEG): container finished" podID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerID="9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3" exitCode=0
Jul 28 17:52:20.032462 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.032358    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerDied","Data":"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"}
Jul 28 17:52:20.032462 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.032388    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"
Jul 28 17:52:20.032462 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.032408    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw" event={"ID":"fb6785bc-a8d9-4d36-b87e-df0bc221bdcc","Type":"ContainerDied","Data":"901c15998dcd316efcd85c00f9caf6772d14b3c2b0df548130e0bac8ac6e52b5"}
Jul 28 17:52:20.038899 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.038875    2552 scope.go:117] "RemoveContainer" containerID="40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"
Jul 28 17:52:20.048344 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048324    2552 scope.go:117] "RemoveContainer" containerID="2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"
Jul 28 17:52:20.048613 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:52:20.048598    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33\": container with ID starting with 2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33 not found: ID does not exist" containerID="2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"
Jul 28 17:52:20.048660 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048622    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33"} err="failed to get container status \"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33\": rpc error: code = NotFound desc = could not find container \"2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33\": container with ID starting with 2ffb4cd5080acfe6f6a1c8e8eda8177f434234bf8f6800ddcbd3f3f2bcb35d33 not found: ID does not exist"
Jul 28 17:52:20.048660 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048640    2552 scope.go:117] "RemoveContainer" containerID="40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"
Jul 28 17:52:20.048847 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:52:20.048824    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f\": container with ID starting with 40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f not found: ID does not exist" containerID="40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"
Jul 28 17:52:20.048912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048856    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f"} err="failed to get container status \"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f\": rpc error: code = NotFound desc = could not find container \"40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f\": container with ID starting with 40e4b47f5510f4c58b3bf0035d8646a3a73bcbccf412aaa17b3a9983dff59a5f not found: ID does not exist"
Jul 28 17:52:20.048912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048879    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:52:20.048912 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.048880    2552 scope.go:117] "RemoveContainer" containerID="46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"
Jul 28 17:52:20.052135 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.052117    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4f50a-predictor-54cf775f94-5gqtw"]
Jul 28 17:52:20.056366 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.056345    2552 scope.go:117] "RemoveContainer" containerID="9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"
Jul 28 17:52:20.064156 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.064130    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:52:20.064426 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.064406    2552 scope.go:117] "RemoveContainer" containerID="46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"
Jul 28 17:52:20.064675 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:52:20.064659    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c\": container with ID starting with 46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c not found: ID does not exist" containerID="46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"
Jul 28 17:52:20.064719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.064682    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c"} err="failed to get container status \"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c\": rpc error: code = NotFound desc = could not find container \"46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c\": container with ID starting with 46c4425a4bf7d04f20ce2e958fa7d0a25f9b2a72034a60331e4c5fba3d86533c not found: ID does not exist"
Jul 28 17:52:20.064719 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.064706    2552 scope.go:117] "RemoveContainer" containerID="9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"
Jul 28 17:52:20.064917 ip-10-0-136-80 kubenswrapper[2552]: E0728 17:52:20.064902    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3\": container with ID starting with 9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3 not found: ID does not exist" containerID="9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"
Jul 28 17:52:20.064957 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.064919    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3"} err="failed to get container status \"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3\": rpc error: code = NotFound desc = could not find container \"9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3\": container with ID starting with 9b529ef0b8906000aa2064516f180946c825f1831bc1d0c42dc70f4f441881c3 not found: ID does not exist"
Jul 28 17:52:20.067891 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:20.067870    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4f50a-predictor-569ccbb779-gwzcj"]
Jul 28 17:52:21.972101 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:21.972068    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" path="/var/lib/kubelet/pods/ea7b2898-7d16-4d1f-85ab-9c12e6e8508f/volumes"
Jul 28 17:52:21.972507 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:52:21.972494    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" path="/var/lib/kubelet/pods/fb6785bc-a8d9-4d36-b87e-df0bc221bdcc/volumes"
Jul 28 17:57:05.359068 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:57:05.359023    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:33.439316 ip-10-0-136-80 kubenswrapper[2552]: I0728 17:58:33.439221    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:30.300785 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.300743    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:30.460330 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.460296    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 18:00:30.460587 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.460543    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container" containerID="cri-o://8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6" gracePeriod=30
Jul 28 18:00:30.460694 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.460585    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kube-rbac-proxy" containerID="cri-o://a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c" gracePeriod=30
Jul 28 18:00:30.585636 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.585610    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 18:00:30.585874 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.585835    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container" containerID="cri-o://f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01" gracePeriod=30
Jul 28 18:00:30.585965 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.585863    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kube-rbac-proxy" containerID="cri-o://0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f" gracePeriod=30
Jul 28 18:00:30.917658 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.917568    2552 generic.go:356] "Generic (PLEG): container finished" podID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerID="0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f" exitCode=2
Jul 28 18:00:30.917658 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.917632    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerDied","Data":"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"}
Jul 28 18:00:30.919271 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.919249    2552 generic.go:356] "Generic (PLEG): container finished" podID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerID="a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c" exitCode=2
Jul 28 18:00:30.919387 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:30.919314    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerDied","Data":"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"}
Jul 28 18:00:31.290150 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:31.290118    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:31.356051 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:31.356022    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:31.600064 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:31.599980    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:31.658710 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:31.658676    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:00:33.425520 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.425499    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 18:00:33.511620 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.511541    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls\") pod \"b6906f40-d78c-436c-bd07-8d8b0ec81237\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") "
Jul 28 18:00:33.511620 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.511592    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config\") pod \"b6906f40-d78c-436c-bd07-8d8b0ec81237\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") "
Jul 28 18:00:33.511832 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.511705    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chljj\" (UniqueName: \"kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj\") pod \"b6906f40-d78c-436c-bd07-8d8b0ec81237\" (UID: \"b6906f40-d78c-436c-bd07-8d8b0ec81237\") "
Jul 28 18:00:33.512023 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.511981    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6906f40-d78c-436c-bd07-8d8b0ec81237" (UID: "b6906f40-d78c-436c-bd07-8d8b0ec81237"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:00:33.513713 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.513687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6906f40-d78c-436c-bd07-8d8b0ec81237" (UID: "b6906f40-d78c-436c-bd07-8d8b0ec81237"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:33.513818 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.513770    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj" (OuterVolumeSpecName: "kube-api-access-chljj") pod "b6906f40-d78c-436c-bd07-8d8b0ec81237" (UID: "b6906f40-d78c-436c-bd07-8d8b0ec81237"). InnerVolumeSpecName "kube-api-access-chljj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:33.533392 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.533374    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 18:00:33.612249 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612229    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") pod \"22f17563-00e2-47d2-91e8-662f3833f2f6\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") "
Jul 28 18:00:33.612375 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612270    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config\") pod \"22f17563-00e2-47d2-91e8-662f3833f2f6\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") "
Jul 28 18:00:33.612375 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612295    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jgv6\" (UniqueName: \"kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6\") pod \"22f17563-00e2-47d2-91e8-662f3833f2f6\" (UID: \"22f17563-00e2-47d2-91e8-662f3833f2f6\") "
Jul 28 18:00:33.612518 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612485    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6906f40-d78c-436c-bd07-8d8b0ec81237-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.612622 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612522    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.612622 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612538    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chljj\" (UniqueName: \"kubernetes.io/projected/b6906f40-d78c-436c-bd07-8d8b0ec81237-kube-api-access-chljj\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.612622 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.612613    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22f17563-00e2-47d2-91e8-662f3833f2f6" (UID: "22f17563-00e2-47d2-91e8-662f3833f2f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:00:33.614284 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.614264    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6" (OuterVolumeSpecName: "kube-api-access-8jgv6") pod "22f17563-00e2-47d2-91e8-662f3833f2f6" (UID: "22f17563-00e2-47d2-91e8-662f3833f2f6"). InnerVolumeSpecName "kube-api-access-8jgv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:33.614339 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.614271    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22f17563-00e2-47d2-91e8-662f3833f2f6" (UID: "22f17563-00e2-47d2-91e8-662f3833f2f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:33.713005 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.712960    2552 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22f17563-00e2-47d2-91e8-662f3833f2f6-proxy-tls\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.713087 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.713012    2552 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.713087 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.713027    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jgv6\" (UniqueName: \"kubernetes.io/projected/22f17563-00e2-47d2-91e8-662f3833f2f6-kube-api-access-8jgv6\") on node \"ip-10-0-136-80.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.932090 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.932014    2552 generic.go:356] "Generic (PLEG): container finished" podID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerID="f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01" exitCode=0
Jul 28 18:00:33.932090 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.932062    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"
Jul 28 18:00:33.932315 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.932092    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerDied","Data":"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"}
Jul 28 18:00:33.932315 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.932159    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4" event={"ID":"22f17563-00e2-47d2-91e8-662f3833f2f6","Type":"ContainerDied","Data":"30a059100090df819f4ec1d45455b34e01880bf45c34d3009c19d413e915e5bd"}
Jul 28 18:00:33.932315 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.932197    2552 scope.go:117] "RemoveContainer" containerID="0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"
Jul 28 18:00:33.933575 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.933554    2552 generic.go:356] "Generic (PLEG): container finished" podID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerID="8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6" exitCode=0
Jul 28 18:00:33.933673 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.933601    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"
Jul 28 18:00:33.933673 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.933622    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerDied","Data":"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"}
Jul 28 18:00:33.933673 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.933667    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf" event={"ID":"b6906f40-d78c-436c-bd07-8d8b0ec81237","Type":"ContainerDied","Data":"a7f582b9047c0f25387f7a4788dba1aa98e41f9bc05c41cfc30cb7d28b8b6331"}
Jul 28 18:00:33.941089 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.941073    2552 scope.go:117] "RemoveContainer" containerID="f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"
Jul 28 18:00:33.948235 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.948220    2552 scope.go:117] "RemoveContainer" containerID="0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"
Jul 28 18:00:33.948505 ip-10-0-136-80 kubenswrapper[2552]: E0728 18:00:33.948489    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f\": container with ID starting with 0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f not found: ID does not exist" containerID="0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"
Jul 28 18:00:33.948554 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.948513    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f"} err="failed to get container status \"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f\": rpc error: code = NotFound desc = could not find container \"0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f\": container with ID starting with 0e7e958d4ea9d31ccadf538b3e2965212f8e2510fbe7379959e3d0a2de43704f not found: ID does not exist"
Jul 28 18:00:33.948554 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.948530    2552 scope.go:117] "RemoveContainer" containerID="f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"
Jul 28 18:00:33.948742 ip-10-0-136-80 kubenswrapper[2552]: E0728 18:00:33.948722    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01\": container with ID starting with f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01 not found: ID does not exist" containerID="f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"
Jul 28 18:00:33.948787 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.948747    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01"} err="failed to get container status \"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01\": rpc error: code = NotFound desc = could not find container \"f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01\": container with ID starting with f581aba5f18fb6313185a87cfc5f3cdd56798f5de9cdd21af5fdc770edfc1c01 not found: ID does not exist"
Jul 28 18:00:33.948787 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.948763    2552 scope.go:117] "RemoveContainer" containerID="a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"
Jul 28 18:00:33.955904 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.955883    2552 scope.go:117] "RemoveContainer" containerID="8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"
Jul 28 18:00:33.956688 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.956672    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 18:00:33.961550 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.961530    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e4209-predictor-595c5d94bb-tg9l4"]
Jul 28 18:00:33.963015 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.962985    2552 scope.go:117] "RemoveContainer" containerID="a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"
Jul 28 18:00:33.963242 ip-10-0-136-80 kubenswrapper[2552]: E0728 18:00:33.963227    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c\": container with ID starting with a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c not found: ID does not exist" containerID="a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"
Jul 28 18:00:33.963295 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.963249    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c"} err="failed to get container status \"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c\": rpc error: code = NotFound desc = could not find container \"a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c\": container with ID starting with a04cd4b07d68d50a973637c7beda5a6b6f9ae9bd6aa4d9b45761801aeb0c833c not found: ID does not exist"
Jul 28 18:00:33.963295 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.963264    2552 scope.go:117] "RemoveContainer" containerID="8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"
Jul 28 18:00:33.963484 ip-10-0-136-80 kubenswrapper[2552]: E0728 18:00:33.963468    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6\": container with ID starting with 8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6 not found: ID does not exist" containerID="8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"
Jul 28 18:00:33.963544 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.963488    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6"} err="failed to get container status \"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6\": rpc error: code = NotFound desc = could not find container \"8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6\": container with ID starting with 8bdf006a6a2e97301dff56d692cc40db294f97225f9311a08098534e2dcef7f6 not found: ID does not exist"
Jul 28 18:00:33.971917 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.971898    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" path="/var/lib/kubelet/pods/22f17563-00e2-47d2-91e8-662f3833f2f6/volumes"
Jul 28 18:00:33.978151 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.975561    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 18:00:33.981778 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:33.981758    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e4209-predictor-5d46777c7f-24bwf"]
Jul 28 18:00:35.971866 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:00:35.971833    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" path="/var/lib/kubelet/pods/b6906f40-d78c-436c-bd07-8d8b0ec81237/volumes"
Jul 28 18:01:02.840872 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:02.840834    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:03.045440 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045408    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ck55s/must-gather-hl7qm"]
Jul 28 18:01:03.045702 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045689    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045703    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045713    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045719    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045727    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kube-rbac-proxy"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045733    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kube-rbac-proxy"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045739    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container"
Jul 28 18:01:03.045750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045744    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045754    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045759    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045765    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045771    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045780    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045786    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045792    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045797    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045804    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045808    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045814    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045819    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045827    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045832    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045838    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045842    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045848    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045853    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045858    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045863    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045869    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045873    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045883    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045888    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045931    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045937    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045944    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045950    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045955    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea7b2898-7d16-4d1f-85ab-9c12e6e8508f" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045962    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045967    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a935516e-f8e9-47d4-94a2-c90e00577ddb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045972    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8174e7bd-0b17-4167-a5f0-28e6398e6f74" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045978    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kserve-container"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.045982    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="22f17563-00e2-47d2-91e8-662f3833f2f6" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046001    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kube-rbac-proxy"
Jul 28 18:01:03.046018 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046007    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5f73a8-86d6-4fb0-8a95-e4362eecd893" containerName="kube-rbac-proxy"
Jul 28 18:01:03.047081 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046013    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kserve-container"
Jul 28 18:01:03.047081 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046018    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6785bc-a8d9-4d36-b87e-df0bc221bdcc" containerName="kserve-container"
Jul 28 18:01:03.047081 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046024    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e48201-4e75-41a0-85b3-f5088eac37cb" containerName="kube-rbac-proxy"
Jul 28 18:01:03.047081 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.046028    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6906f40-d78c-436c-bd07-8d8b0ec81237" containerName="kserve-container"
Jul 28 18:01:03.048934 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.048917    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.052330 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.052310    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ck55s\"/\"kube-root-ca.crt\""
Jul 28 18:01:03.052456 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.052337    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ck55s\"/\"default-dockercfg-p6qwz\""
Jul 28 18:01:03.052868 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.052853    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ck55s\"/\"openshift-service-ca.crt\""
Jul 28 18:01:03.058526 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.058505    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ck55s/must-gather-hl7qm"]
Jul 28 18:01:03.061900 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.061874    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:03.134317 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.134242    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9x45\" (UniqueName: \"kubernetes.io/projected/7bd2ca63-f1be-47b8-9668-37252814cc75-kube-api-access-f9x45\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.134317 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.134291    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bd2ca63-f1be-47b8-9668-37252814cc75-must-gather-output\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.154707 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.154684    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:03.235625 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.235599    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9x45\" (UniqueName: \"kubernetes.io/projected/7bd2ca63-f1be-47b8-9668-37252814cc75-kube-api-access-f9x45\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.235742 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.235649    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bd2ca63-f1be-47b8-9668-37252814cc75-must-gather-output\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.235949 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.235934    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7bd2ca63-f1be-47b8-9668-37252814cc75-must-gather-output\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.249897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.249877    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9x45\" (UniqueName: \"kubernetes.io/projected/7bd2ca63-f1be-47b8-9668-37252814cc75-kube-api-access-f9x45\") pod \"must-gather-hl7qm\" (UID: \"7bd2ca63-f1be-47b8-9668-37252814cc75\") " pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.357977 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.357949    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ck55s/must-gather-hl7qm"
Jul 28 18:01:03.484286 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.484262    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ck55s/must-gather-hl7qm"]
Jul 28 18:01:03.486056 ip-10-0-136-80 kubenswrapper[2552]: W0728 18:01:03.486028    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd2ca63_f1be_47b8_9668_37252814cc75.slice/crio-ef63b2008507bf995cad9bf2fda0d691d13ed4768885d4d9145e7be33090983a WatchSource:0}: Error finding container ef63b2008507bf995cad9bf2fda0d691d13ed4768885d4d9145e7be33090983a: Status 404 returned error can't find the container with id ef63b2008507bf995cad9bf2fda0d691d13ed4768885d4d9145e7be33090983a
Jul 28 18:01:03.487888 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:03.487872    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:01:04.044689 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:04.044661    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ck55s/must-gather-hl7qm" event={"ID":"7bd2ca63-f1be-47b8-9668-37252814cc75","Type":"ContainerStarted","Data":"ef63b2008507bf995cad9bf2fda0d691d13ed4768885d4d9145e7be33090983a"}
Jul 28 18:01:05.050902 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:05.050866    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ck55s/must-gather-hl7qm" event={"ID":"7bd2ca63-f1be-47b8-9668-37252814cc75","Type":"ContainerStarted","Data":"01d8199d958ec962bde16713d6826666e3982f8de945620e86aeeafd16a49cb2"}
Jul 28 18:01:05.051468 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:05.050918    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ck55s/must-gather-hl7qm" event={"ID":"7bd2ca63-f1be-47b8-9668-37252814cc75","Type":"ContainerStarted","Data":"612ad62a3f82b1fe015376e2660f1289ea4c0714ac61851ea78772384da107f0"}
Jul 28 18:01:05.488123 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:05.488083    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pdtks_51709ba2-21ed-49d5-8a4f-f4c2b1f92745/global-pull-secret-syncer/0.log"
Jul 28 18:01:05.594480 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:05.594444    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log"
Jul 28 18:01:05.731610 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:05.731582    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-80.ec2.internal_eb9f2dabc625c037e2ff47a5be458f87/haproxy/0.log"
Jul 28 18:01:08.637428 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:08.637397    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:08.701344 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:08.701315    2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kq74_6c14d132-50ec-45ad-9967-87f75f5d4730/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:09.649905 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.649878    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/alertmanager/0.log"
Jul 28 18:01:09.685840 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.685807    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/config-reloader/0.log"
Jul 28 18:01:09.717736 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.717710    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/kube-rbac-proxy-web/0.log"
Jul 28 18:01:09.751179 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.751141    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/kube-rbac-proxy/0.log"
Jul 28 18:01:09.781562 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.781528    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/kube-rbac-proxy-metric/0.log"
Jul 28 18:01:09.812649 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.812615    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/prom-label-proxy/0.log"
Jul 28 18:01:09.844161 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:09.844133    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9f290579-5839-4bf9-8876-1673e4f3c347/init-config-reloader/0.log"
Jul 28 18:01:10.079268 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.079236    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-6tzs4_0837ee53-b9ed-4ff2-93e7-f272e30fcb82/monitoring-plugin/0.log"
Jul 28 18:01:10.324005 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.323961    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmzxb_383f0e38-a7c2-4981-803c-e89f3258efcd/node-exporter/0.log"
Jul 28 18:01:10.350798 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.350721    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmzxb_383f0e38-a7c2-4981-803c-e89f3258efcd/kube-rbac-proxy/0.log"
Jul 28 18:01:10.380830 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.380806    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmzxb_383f0e38-a7c2-4981-803c-e89f3258efcd/init-textfile/0.log"
Jul 28 18:01:10.851888 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.851807    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5cd568-hlwqg_9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd/telemeter-client/0.log"
Jul 28 18:01:10.881279 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.881251    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5cd568-hlwqg_9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd/reload/0.log"
Jul 28 18:01:10.917893 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:10.917859    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c5cd568-hlwqg_9fa41006-46c0-4d9c-bb9f-e3dd8362b9dd/kube-rbac-proxy/0.log"
Jul 28 18:01:13.464410 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.464345    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ck55s/must-gather-hl7qm" podStartSLOduration=9.876671978 podStartE2EDuration="10.46429448s" totalImagesPullingTime="587.622502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:05.070878563 +0000 UTC m=+3045.623188478" watchObservedRunningTime="2026-07-28 18:01:13.46429448 +0000 UTC m=+3054.016604375"
Jul 28 18:01:13.467549 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.467517    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"]
Jul 28 18:01:13.473228 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.473208    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.482062 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.482037    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"]
Jul 28 18:01:13.522897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.522406    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-podres\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.522897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.522459    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-sys\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.522897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.522547    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-lib-modules\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.522897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.522596    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-proc\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.522897 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.522646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbfn\" (UniqueName: \"kubernetes.io/projected/30afa6e4-73a1-4501-8b14-e53decb6b242-kube-api-access-7gbfn\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.623825 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.623792    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-lib-modules\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624113 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-proc\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624279 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbfn\" (UniqueName: \"kubernetes.io/projected/30afa6e4-73a1-4501-8b14-e53decb6b242-kube-api-access-7gbfn\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624540 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624151    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-proc\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624675 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624078    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-lib-modules\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624675 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624517    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-podres\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624675 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624593    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-sys\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624675 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624602    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-podres\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.624949 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.624681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30afa6e4-73a1-4501-8b14-e53decb6b242-sys\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.635138 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.635091    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbfn\" (UniqueName: \"kubernetes.io/projected/30afa6e4-73a1-4501-8b14-e53decb6b242-kube-api-access-7gbfn\") pod \"perf-node-gather-daemonset-mdvx6\" (UID: \"30afa6e4-73a1-4501-8b14-e53decb6b242\") " pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.787781 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.787707    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:13.954777 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:13.954751    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"]
Jul 28 18:01:14.099424 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:14.099383    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6" event={"ID":"30afa6e4-73a1-4501-8b14-e53decb6b242","Type":"ContainerStarted","Data":"8362443436b9725d988490ce911ea67e131b4afa5efaff37491373a0c09dfa65"}
Jul 28 18:01:14.099581 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:14.099449    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6" event={"ID":"30afa6e4-73a1-4501-8b14-e53decb6b242","Type":"ContainerStarted","Data":"681f1f5206fda74165f3e32e8e06e5e4e39fb5f453c1637b4289995934c202dc"}
Jul 28 18:01:14.099581 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:14.099478    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:14.116931 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:14.116881    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6" podStartSLOduration=1.116866294 podStartE2EDuration="1.116866294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:14.116312472 +0000 UTC m=+3054.668622375" watchObservedRunningTime="2026-07-28 18:01:14.116866294 +0000 UTC m=+3054.669176184"
Jul 28 18:01:15.020056 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:15.020024    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nspz_2019a327-9b47-402f-9579-4015f8476743/dns/0.log"
Jul 28 18:01:15.048070 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:15.048045    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nspz_2019a327-9b47-402f-9579-4015f8476743/kube-rbac-proxy/0.log"
Jul 28 18:01:15.214788 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:15.214754    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dx6mc_09870d81-7076-49f2-a276-6530e16cf472/dns-node-resolver/0.log"
Jul 28 18:01:15.722959 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:15.722927    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d5779b67c-555dc_390ef382-926e-4dde-a1c5-874ccf6cdf93/registry/0.log"
Jul 28 18:01:15.793542 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:15.793518    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l8bj2_b44d39e4-6f0e-43ba-962f-14f740a7bbfe/node-ca/0.log"
Jul 28 18:01:16.951497 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:16.951473    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cm88p_ed188f2a-8cb8-47dd-9389-e46104e0de0b/serve-healthcheck-canary/0.log"
Jul 28 18:01:17.438916 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:17.438881    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22wdg_f763302e-899d-450b-a3fb-4e29fc13f2ff/kube-rbac-proxy/0.log"
Jul 28 18:01:17.464724 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:17.464693    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22wdg_f763302e-899d-450b-a3fb-4e29fc13f2ff/exporter/0.log"
Jul 28 18:01:17.491290 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:17.491267    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-22wdg_f763302e-899d-450b-a3fb-4e29fc13f2ff/extractor/0.log"
Jul 28 18:01:19.880785 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:19.880750    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cd4d44c67-kj262_2abee3f1-749d-427f-a164-88293c7580d9/manager/0.log"
Jul 28 18:01:19.937771 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:19.937742    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-c2v7k_533ea5cb-5ec5-45d5-9a90-9b2e5293903f/server/0.log"
Jul 28 18:01:20.048879 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:20.048849    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-rxtkg_2c2e47ce-75c3-469e-a024-1566394c40f8/manager/0.log"
Jul 28 18:01:20.115106 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:20.115083    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ck55s/perf-node-gather-daemonset-mdvx6"
Jul 28 18:01:20.164623 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:20.164556    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-pv5cz_f986c8a2-276f-4ced-971f-fcbc5968d76a/seaweedfs/0.log"
Jul 28 18:01:24.571304 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:24.571272    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gj2pm_f81d3091-4ca9-47a1-a738-2ba312ef1ed6/migrator/0.log"
Jul 28 18:01:24.610940 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:24.610909    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gj2pm_f81d3091-4ca9-47a1-a738-2ba312ef1ed6/graceful-termination/0.log"
Jul 28 18:01:26.272596 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.272570    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:01:26.298722 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.298695    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/egress-router-binary-copy/0.log"
Jul 28 18:01:26.325643 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.325619    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/cni-plugins/0.log"
Jul 28 18:01:26.352798 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.352779    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/bond-cni-plugin/0.log"
Jul 28 18:01:26.379540 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.379520    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/routeoverride-cni/0.log"
Jul 28 18:01:26.405926 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.405898    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/whereabouts-cni-bincopy/0.log"
Jul 28 18:01:26.433608 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.433581    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zblqh_4d386b9a-93a0-4b9d-b3cd-1bd3c0e89963/whereabouts-cni/0.log"
Jul 28 18:01:26.775315 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.775287    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pscn6_1df21e2b-1c50-4d77-8985-2aa6b8a75c2a/kube-multus/0.log"
Jul 28 18:01:26.964307 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:26.964276    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dww89_52ed84b2-9963-498c-8594-54462ee49011/network-metrics-daemon/0.log"
Jul 28 18:01:27.008126 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:27.008098    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dww89_52ed84b2-9963-498c-8594-54462ee49011/kube-rbac-proxy/0.log"
Jul 28 18:01:28.281750 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.281716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/ovn-controller/0.log"
Jul 28 18:01:28.318835 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.318805    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/ovn-acl-logging/0.log"
Jul 28 18:01:28.371247 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.371224    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/kube-rbac-proxy-node/0.log"
Jul 28 18:01:28.417610 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.417586    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:01:28.467193 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.467163    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/northd/0.log"
Jul 28 18:01:28.515189 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.515167    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/nbdb/0.log"
Jul 28 18:01:28.562009 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.561924    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/sbdb/0.log"
Jul 28 18:01:28.634017 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:28.633978    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6lk7_8bf73946-55d9-4630-8bc2-f6c27634a26c/ovnkube-controller/0.log"
Jul 28 18:01:30.173901 ip-10-0-136-80 kubenswrapper[2552]: I0728 18:01:30.173872    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-g4wzj_3d1de778-9e38-4e7d-927e-6d8b4d89878a/network-check-target-container/0.log"